Because you're perverse and wrong and impure and should set yourself on fire to purge yourself of the evil.

I kid, I kid. I joke, I joke. Perhaps it's because they've taken a liking to your clothing you feel you have something in common and are thusly attracted. Maybe you are turned on by the fact that you'll be wearing clothes that they've worn, giving to a kind of second-hand intimacy. Or you could just have a fetish about this sort of thing. Each unto their own.