When he comes back with "I understand you just can't fit a relationship into your life right now..." feel free to tell him "It's not me, it's you."
I held back when I wanted to give someone that response once, and he went on to be a jerk to one of my friends, I later found out. Now I wish I had just said it instead of trying to be nice.
The pews never miss a sermon but that doesn't get them one step closer to Heaven.
But at least the pews never attend yoga!