Here's my theory: Maybe it's not that you're displeasing to look at, but the contrary. People generally like to look at what's pleasing to the eye. It's nothing bad, they don't mean to insult you (rather the contrary). It's human nature.