As Laddy mentioned, those who use magic are seen as more intellectual and submissive compared to warriors who are seen as more physical and dominating. The truth is that those who wield arcane arts are no less capable of physical activity than those who wield martial arts, but this may be done as something similar to gameplay balance rather than storyline balance.