I'm pretty sure that you only fight him twice, and when you reach him the third time he calls the guards over to fight you.