It's mostly story/gameplay segregation, it would be really obnoxious if the game forced you to stick with the party you had in these instances cause what happens if you brought a character that was under-leveled or in other games, try to move forward without the proper party member.