Agreed. Clobbering ogres for fun and profit. I end up making up some small story on why the light warriors are wandering aimlessly instead of trying to stop Garland.
But it is not just the gold. It's the leveling. Well worth it. Not to mention that a short boat ride away and you can pay inn prices from the start of the game.
Besides, you just need to level enough to get to through the game to the giant's hallway.