Maybe his Brave was too low, or his Faith was too high. If you get them past a certain point your character will leave. The enemy could've Invited him too, which automatically takes him out of the party. Did you save after he left the party? Whether it was a bug or not, if you saved he'll never come back.