You will have to use them, yes, but it doesn't matter what their levels are. The enemy's levels are based on your character's levels, so the difficulty will adjust with the party. Just switch your junctions over and you'll be fine.