This is the first time I ever heard someone say that IX is hard, maybe a secret boss in it, but generaly mostly everyone could agree that it's one of the easiest.
Out of the ones I've played IV gave me the most trouble, I would have said XII instead, but most of my deaths in that game came from non- story events. The game is challenging, but I love it.