Ruby and Emerald can be tough, but they just take a lot of preparation to be able to beat.
And I always get more irritated by Ruby. Emerald has more HP, but you can't just start up the battle with a fully healed and prepared party against Ruby. That, and the fact that he likes to counter Knights of the Round and break your rythm makes him a tougher opponent, I think.