I think it has more to do with the fact that they are the most prominent Every game has characters I like and dislike. If I can just ignore the ones that I don't like then they don't bother me, but if they happen to be the main character it just lets the resentment build up.