If you're playing the Zodiac Age, they did dial the difficulty back, which was my one major gripe with the game. To answer your question, there isn't really a spike in difficulty as much as it's easier for you to hit up difficult optional areas much earlier than you need to, so that can offer a bit of challenge. Hunts are definitely more challenging than the story bosses, as well as the optional Esper battles so you should definitely tackle them, especially since they were designed to be dealt with as you played and not simply saved for endgame.

If that's not working for you, then perhaps turn this into a challenge run I guess.