The way the Tower of Babil is described, it sounds like it stretches all the way to the moon, so the Giant would come down to Earth directly from there. But since there's no sign of the Tower on the moon itself, I guess it falls short.

Since the Enterprise rises off the top of the screen when approaching the Tower of Zot, it's presumably floating in the air somewhere, like the Flying Castle from FFI.