That cave had some of the most ridiculus random encounters in the entire game, and it was annoyingly long if you wanted to collect all the treasures. I feel it was difficult enough without a boss at the end.
I agree that it was annoying long to get all the treasures, but there's a difference between random encounters and a boss fight, no matter how tough they are.