To Kill A Raven, Final Fantasy XIV. I'm sure if I had tried it that the hard version of this same fight would be my choice. :p It wasn't actually terribly annoying but it certainly was annoying how you would spend 20-25 minutes before the real test of whether or not you will win the fight. MMO boss fights can be a real bitch sometimes. But the fight was interesting to master, I suppose.
I can only imagine the most annoying boss fight ever would be a tedious, monotonous, repetitive task relying on quick-time events that are questionable at best over their timing and the fight lasts over 30 minutes. If such a boss fight as this exists, I would call it the most annoying boss fight ever. It's pretty much everything I wouldn't like. Oh, wait. Make it force you to partner up with a different person every time, a la Journey. You can't talk with them, you don't know who they are, but you're relying on them as well as yourself now! And the boss would be standard for most of the fight but right at the end would do a move that often will one-shot you, and it won't give any warnings as to when this happens. No HP bar. They could market this game as The Most Annoying Boss Fight Ever, now that I think about it. One achievement: Win.