If you are going to care what others are going to think, then yes, it could be a problem. The point is not to care about any associations; the less a deal you make of it when someone brings it up, the less of an issue it is.

If the question comes up about what an employer may think: why would you work for someone that ignorant? And the suppositions should stop there.