You can't change the course of the game by telling Banon that you won't help out, and venturing back won't accomplish anything for you, however, if you tell Banon no three times in a row the story will progress like normal, except you will receive a Genji Glove which is an awesome relic, especially at this point in the game.