Tough call, but I've not really played any of the earlier ones (which are probably more the correct answer for this). But out of the ones I've played, probably X. Now the game itself is really quite easy, but the post-game content is some of the hardest, most inane sack of crap I have ever experienced. All the irritating mini-games the game forces you to do if you want the celestial weapons, and the superbosses are really tough.
LR:XIII is also pretty tough. I played on easy and some of the bosses were still very technically demanding (Aeronite and Ereshkigal, but still).