I'm just guessing you meant the party members who leave your party at some point in the game. If so, then yes, you must wait until you defeat the Giant of Babil, not the Tower. After that you can switch between your party members between your current and all other formers except Tellah. They are also all leveled to be the same as your current ones.