I try and use everyone, and to be honest it led to me being quite overpowered with doing little to no real grinding. Just following the story. Definitely worthwhile, and doesn't take that much longer.