A lot of people do like to bring up that argument. To answer it, I give you a quote from Kuja himself:

Kuja: "It's the original crystal... This is where it all began..The birthplace of all things..Once I destroy it, everything will be gone."Gaia, Terra, the universe, everything.. Your lives and memories, too, of course. HA-HA-HA-HA-HA-HA-HA!"

And then Necron:

""Kuja was a victim of his own fear. He concluded he could only save himself by destroying the origin of all things - the crystal."