You need Alicia and Lavian in the party to acquire a unique accessory later in the game. (No quest, just to be able to buy the thing). The accessory is extremely good, but in no means necessary.
Even if you don't want someone it is always better to accept them, de-equip all of their stuff, and then dismiss them,

As for other characters that you get from now on, they are all special so I would hold on to them even if you never use them.