These guys are really interesting. As far as I can tell, they're outcasts from Yevon. Mushroom Rock Road was obviously a big of a black mark on their record thanks to working closely with Al Bhed, but Wakka talks about how Chappu took a machina weapon as opposed to Brotherhood, so there's a history of them "betraying Yevon."
So why are the numbers so populous?
I get the "defending your families" thing, but when it seems like more people are casting Yevon aside to protect their Yevon-following families, you'd think the institution would crumble. But, because summoners are the only way to truly defeat Sin, that makes Yevon strong, strong enough, surely, to make the Crusaders unnecessary or discouraged or something.
It seems to be a weird status quo.