It is very infuriating when people are like that!! A lot of times, I think it's a bit of jealousy, and the other times it's a lot of ignorance. It's true that saying - Don't judge a man/woman until you walk a mile in their shoes.
I used to get a lot of grief from my dad. He always called me too skinny, unhealthy, just eat whatever you want. It really hurt me deeply. People can be so cruel.