Chocobo race in FFX. Butterfly catching in FFX. Lightning Dodging in FFX, if that counts. Sphere Break in X2. Blitzball. Yeah, FFX wins for annoying mini games.
The "Jobs" in Kingdom Hearts 2 were annoying too.