I usually give the fourth character some generic weapons or spells that I've bought from the shops or keep their default equipment. If I've found a special weapon for them, the other characters usually aren't using that weapon type, so most likely they can keep it. For example, Gordon and Richard were the only ones using spears in my game, so I let them keep them even when they left the party. That's why, I didn't really mind them keeping the equipment. They're helpful in Souls of Rebirth extra dungeon anyway.