The most annoying would be the Zozo clock puzzle in FFVI. It's easy enough to do, but you're constantly interrupted by random battles.

The hardest would be Excalibur II, Lightning-dodging, or the Chocobo racing from FFX.