Just keep on playing. I also got bothered by that part, but it's worth it.
If you have problems, just run away from the encounters or make your party three black mages and a red one.