I think a big reason is that it is very hard to even conceive of a situation as being us vs. them when you are living with them, sleeping with them, raising children with them and in love with them. That kind of relationship is unique to the genders, and not the case for any kind of ethnic minority.
Because men and women have that kind of intimate relationship it makes it easier for the representatives of the patriarchy to paint feminists with such an ugly brush. They say feminists are _____ (fill in the blank with an ugly stereotype). If you as a woman have an emotional investment in the way a man perceives you, you don't want him to see you as that ugly stereotype, so you distance yourself from feminism. You see that dynamic at play in other types of oppression (like the whole "good hair" and light skin being desirable among blacks because it is the most acceptable to whites) but nowhere near the same degree because as a whole, men and women are much more interrelated than blacks and whites.
To Trenell, MizKerri and geeky:
I pray none of you ever has to live in a communist state.
Geeky is my hero. She's the true badass. The badass who doesn't even need to be a badass. There aren't enough O's in cool to describe her.