Although the FF series holds promise of many annoying mini-games, my money's on Suikoden IV:

The Mouse Catching game is annoying for two reasons: the mice are hard to pick up, and if Champo collides with a wall or crates, he falls over stunned for about five seconds. So if you hit a wall even once there's no way to win on the harder levels.

Although the Lighting Dodging game in FFX gets an honorable mention. It's an endurance ordeal, and if you mess up even once, its over. Not fun at all.

Some have an amazingly steep difficulty curve like fishing in FFXII or Lightning Towers in FFX-2, where the easier levels are simple and the harder ones are nearly impossible. But at least they are fun some of the time.