I'd say the numbers are so populous just because they are actively doing stuff to try to ward of Sin and protect the citizens of Spira. Perhaps a part of those people think during the multitude of years that Sin is attacking that maybe there is a better way to stop Sin, or maybe they are just doing their own thing to not only bring protection to the citizens, but also allow for more safe passage for the Summoners so that the Calm may be able to come about quicker. The summoners (and thus Yevon) have a lot of power just because the only way the Calm can come about is through the Final Aeon; however, as has been mentioned in another thread, staggeringly few Summoners have ever been successful in this quest. Because of that success, people must give support to Yevon, but I'm sure they want to be doing something as well to find another way to defeat Sin. If Summoners would become obsolete in fending off Sin, I'm sure Yevon would have lost a lot of support.

I really need to play X and examine the Crusaders more closely once the HD collection comes out. I've always been pretty interested in how they fit into the world of Spira.