I seem to recall the pyramid in FFV being especially perilous, but I can't remember exactly why. Aren't you short a member during it as well?

FFIII Crystal Tower wasn't super hard on my run, just stressful.