I usually only increase the stats they're good at as long as they don't have a need for survivability. If that's not the case then I'll throw a few at defense or HP every now and then. I like having my specialists.