If you're using Switch in the Junction menu, then all magic switches, not just Junctioned magic. If you want to say, move spells from your inactive party members to your active ones so that your inactive party members have no magic, go to the Magic menu instead. You can move them one spell at a time, or have one party member give/take all spells to/from another.