I know where you're coming from, friend, my PS2 controller is as knacked as an old man's hip.

On topic; I was around level 14 when I fought the unknown judge, and I believe Vossler was at a comfortable level 17. Make sure that you target only one of the judges, close down on him with the whole party, and then systematically move on to the next judge. The other lower ranked imperials will automatically depart from the battle once all judges are killed, and make sure you heal regularly but attack as well. Do not think about making a white mage who will sit in the corner casting healing spells, for she will die, and you need all the attack power you can get.

Good luck, and no more breaking controllers!