For unimportant chitchat, don't feel you need to respond right away. Slow down the pace of your responses to a couple of hours, instead of a couple of minutes.

When you see him just tell him not to take it personally but...(whatever your reason is).