Yea, that is pretty annoying. But, I like how in Golden Sun: The Lost Age, if you keep saying No, the Old Man calls you out on it and says something like, "Disagree all you want, but you're doing it." And of course at the start of the first game, saying No actually means No and you don't go on an adventure and the game ends.