X5 and X6 rank at the bottom of my Mega Man list. I don't like either. I don't like X7 either, but it isn't as bad as X5 or 6.
What gets really annoying in X6 is fighting the High Max, especially if you can't harm him.
And the endgame levels are so incredibly tough. Right off the bat in the first castle stage you're going to find a wall you cannot scale unless you've picked up the armor upgrade. The second and third castle bosses are no fun at all.
But don't worry, Sigma is a piece of cake in X6. Probably the easiest one in the series.