The "And would not let you go" is reference to the circumstances you meet the answer in regards to line 2 and 3

But unfortunately you're wrong.

You'll have to look more carefully at the amount of times you meet this, because it is slightly a trick sentence with the twice protector, three times a foe.

And when I say met royalty, I mean face to face, hence me saying its literal. Not present in the sense of being in the same area.