I agree. But I think what ends up happening is that hair, like other singular physical features, can be symbolic, and that's where ppl run into trouble sometimes IMO.
Taking hair as an example: Depending on social context, it can mean anything from "power executive" to "political [or ethnic] outcast". It's all in the meaning that's assigned to it. At times, hair is mostly just viewed as an appendage of the skin atop the scalp. In other contexts, a desired hair type is someone's fetish, and is granted disproportionately high value relative to the person wearing it. When a man or woman fetishizes hair, or a body part etc., suddenly that person's value is reduced to a desired object. If someone fetishizes hair you don't have, you'll know because they will make repeated, disparaging comments, including criticizing you in an attempt to make you change your hair to the type they fetishize.
The bigger problem with a fetishist is that s/he only knows how to fixate on one thing as a way to distract her/himself from an intense fear of intimacy with another human being. If someone ever does reject you outright "because of your hair", suspect that they were going to find something about you to reject anyway because they're just not ready to commit.
That said, I think most ppl just have preferences, sometimes strong ones.