Here's the crucial difference. In an RPG, to swing your sword, you press a button to select "Swing Sword" from a menu. In an adventure game, to swing your sword, your press a button that always causes your hero to swing his sword, regardless of context. The Legend of Zelda series is a series of adventure games.