Learn "Toad Song" in the tower, then cast it, then cast Mute on him. He'll be a pushover after that!