Adray is inevitable. No matter what you do, you get him.
The optional characters in the game are: Nel, Albel, Peppita and Roger.
To get Albel, you have to talk to him in the Kirlsa inn while he is still in your party, and for Nel, I don't remember, cause I always got her by default. I don't think it's possible to get Nel if she didn't come at the top of the training facility, right before you left the planet or whatever.