I'd seriously consider diffusing. How is your hair for 2nd day hair? could you wash the night before? I had that same problem and I started doing the spritz and con dish method. I dont know why, but when I do that, my hair dries way faster.
Spritz and Condish Washing for Porous Types+Thirsty Curls Everywhere!