My wives only left me after I did the gender bender quest. I could marry and bed as many women as I wanted but when I traded my sword for a skirt all the gifts in the world couldn't keep them happy.