It doesn't take any longer to dry. I like adding olive oil as needed to control frizz, just a drop or two at a time. I apply it when my hair is dry, not because it takes longer to dry, but because it's easier to tell when I've used enough. I don't find that oil cures my frizzies, just reduces it a bit and reduces tangles.
I'm having the same problem as you with CO though. My hair is curlier, but it feels stiff and full of product. What is your hair like? Porosity, thickness, elasticity? A different conditioner with different ingredients may work better. I think maybe the protein in my conditioner wasn't agreeing with my hair.
My hair is porous, medium thickness, good elasticity, medium density, 2c/3a, APL, dark brown with a silver lining. Biggest enemy: frizz!