There's four refs in football. Just becuase most fouls occur on the ball that's not to say there isn't a hundred other things to watch out for. Offside for example. Probably the hardest rule to referee in any sport, anywhere, especially since FIFA's new ruling, which confuses it even more.

Football is not an easy game to officiate. Controversial maybe, but I actually believe that Basketball and Football have a lot in common. Certainly more in common than Football and American Football.