Well what about all those statues in Remiem temple? If we go by the statue logic, there would be at least 10-20 high summoners.

I just believe only 5 were named because that's what the story demanded. I believe it was implied there was a lot more.