Saria, does he slur his words a lot? Speak very quickly? Spatter his speech with long pauses wherein all he says is "eh..............."? It helps so much to watch the mouth as a French person speaks. I hope it gets easier in person.
Don't forget to purse your lips when you reply. He should start nodding in understanding once he sees you making the French-talking face. Besides, you know all his terms from your training. Give yourself that moment in your mind to put a picture to his words.
I always seem slow-witted with native French speakers because my father spoke more slowly, and because I spend time imagining their words as pictures in my mind (at least for a while).