According to the manual, there are both genders, but only the females ever go out. It's that simple.

Personally, I'd be pissed if they ever made Mithra males available, because then my current Mithra character would be wasted--The only reason I'm playing as a girl at all is because I have no choice.