When killing those harder bosses, you only use certain skills anyway (like Bum Rush). I've never had to "really" rely upon magic at any point in the game, except for Fanatics Tower of course.