I don't get why everyone is going on about it to be honest. If the show was called "The Simon Cowell Show" no one would really care.
In the X-Factor a lot of people that appear on the show don't have what they call the "X-Factor" but they still get to appear.

All that said, I dislike the show and don't think I've ever sat through even a single episode all the way through.