You need to go back and forth from 600AD and the present to repair the castle. Get some tools from the present I believe, go back and give them to the guy's ancestor, who will start repairs on the castle for a price. Once the castle is all repaired, Frog can pay his respects and you wont have to fight anyone.