Twice, 'cause you fight him once in Traverse as Guard Armour, then once as Raverse Armour, and all the other times you can fight are entirely optional!