Well, I was at level 43 with all six of my characters when I fought him, so it wasn't much of a problem for me. However, I recommend taking out all of his minions first, then casting haste on everybody, and pummel him with physical attacks. If it's still too much of a bother, I would recommend you level up your other three party members just a bit.