I wonder if it's too dry by the time you try to distribute the mousse. If I try to apply something by then the hair sticks a little and messes up clumping unless I wet hand it in. Your curls might be strong enough too to take it in when it's pretty wet and then towel. Also ( and I'm still pretty new only doing this about 5 weeks) I discovered I kind of death scrunch in product or with towel. If I scrunch much more gently, slower and hold to give the towel a chance to suck up water it disturbs things less. Instead of scrunch scrunch scrunch it's more like scruuunch hold. Scruuunch hold. If that makes sense. Your goal is to do as little disturbing as possible.
You can see my wavy tutorials here: https://www.youtube.com/playlist?lis...ZNxhBHSXAW40OQ and my wavy blog is The Wavy Nation http://wavynation.wordpress.com