I've never heard you complain when people used that phrase on you, and if you don't complain no one is ever going to know it bothers you. If they don't know, they're gonna keep saying it and you can't get mad at them for that.

I generally think that most people are nice enough to refrain from saying something that really upsets you.