I'm surprised that your low porosity hair takes a long time to dry, unless you have a ton of it. My hair is highly porous and that is usally an issue with porous hair.
As far as will it take less time to dry if you skipped it? Probably not. Would it keep your hair from allowing the moisture in? Not if you have started off with warmer water to open the cuticle to allow the moisture in to begin with. Will it add shine? Definitely.
All in all, it's up to you.
eta the reason that the cold water rinse reduces frizz is that it does lay down the cuticle. That's also how it adds shine.