I can only speak for myself, but for me it has a lot to do with the tone and the intent of the question. That and the person's history with having been asked this question.
For instance, I grew up in Baltimore, which is largely black with some white people. We rarely ever saw anyone of any other race, so when I was growing up and kids said to me, "What are you?," they didn't mean it in a nice way. They meant why do you look different from every other person I've ever seen. And that hurt.
Nowadays, when someone says, "What's your ethnic background?", that doesn't bother me much. Like you say, they are mostly just curious. I actually find my ethnic ambiguity and the combinations I've been asked funny.