Yeah, that's when it gets annoying. I actually met someone who claimed to be so Japanese that she would "accidentally" use 'r' sounds instead of 'w' sounds and stuff. She was full of crap, though.
Words like anime and katana are used to actually distinguish between two similar things, though. They have their use.