No the point of Swap is that if one character is low on HP & MP they can easily refill both. It just happens to have a nifty side effect of boosting stats. I don't use it myself, though.