I think Final Fantasy IX and Final Fantasy XI get more crap for no legitimate reason than any other titles. That's not say there aren't good reasons to dislike the games, but they just seem to get a lot more shallow complaints than other titles. Usually when I hear someone complain about other titles in the series, they can present a good reason for such an opinion.
Vivisteiner presented the reasons I might agree that FFIX deserves the most unreasonably disliked FF title, but having played FFXI, I truly believe that game gets more crap than any other simply because it's an online game.
Don't get me wrong, there are plenty of reasons for anyone to detest Final Fantasy XI. Most of these reasons, however, are only discovered after playing the game for some time. Still, I hear more people bitch and moan about FFXI simply because it's an online game. The complaints have been going around since the game was announced as an MMORPG. I don't consider it unreasonable to not wish to play FFXI because it's online (online games aren't for everyone, after all), but there's no reason to go on about it being such a bad game just because of that.
As for the other titles, there are legitimate reasons for people to dislike the games. Final Fantasy I isn't exactly what someone who was introduced to RPGs with FFVII might expect, and that would naturally put them off of it. FFII's battle system is also good reason to shy away from that game. I never heard that FFIII DS got a lot of crap, but I never paid much attention to what people were saying about that anyway. Many people, myself included, dislike FFVIII because we're unsatisfied with the plot for many reasons (most of which are irrelevant to this thread anyway). FFTA was a disappointment to many because it's not what we expected for a semi-sequel to Tactics. As far as Crystal Chronicles goes, I've not enough experience with the game (read: never played it) to know if there's real justification.
I don't see many people blindly hating any of these games because of superficial details like I do with FFIX and FFXI, so I can't say they're the most unreasonably disliked FF titles. I might say that FFVIII gets more crap than any other game, but I wouldn't say that's entirely unreasonable.