I've been using MO all summer. I only CO wash and I don't have any buildup at all.
I think part of the reason that many think MO is the devil is because it is often used as an ingredient along with lanolin, and wax in products. How would you know what is causing build-up? Would it be lanolin, wax, or mineral oil? - so people just starting avoiding all those perceived bad products.
I can use a small amount - about 12 drops (actually, I'm using 18 now) on my waist length hair, and it does the trick of moisturizing and smoothing very well, and it also comes out cleanly with CO wash. Trust me, you won't suffer any ill effects, and you just might like it.