It'd be in your best interest to level all of them, I was just lazy. I'm not sure if you even lose out on exp (it's been a while) and switching doesn't cost anything.

If you're lazy like me, or REALLY dislike a certain character then don't bother. Otherwise, go for it.