I'm in four different martial arts programs (five if you count Kendo as a martial art, which you shouldn't) as well as fencing. And I've never actually lost a fight, other than one time in California where I tried to take on four different guys at once because they were trying to mug/rape some girl in an alleyway. But I dislocated one of those guy's shoulders and broke another guy's nose before the other two tackled me at the same time.
So people that know me are too afraid of me to fight me, and people that don't are a little put off by my demeanor... I guess. I'm not an intimidating person, but something about me scares people.

Also, some sound advice from everyone's favorite source: