He was better in the later films, and I don't necessarily hate him. It's that thing when people are obsessed with source material (like I am in this case!) that too much of a departure is considered sacrilegious. I'm sure that many Harry Potter fans that have only seen the films love Gambon's portrayal. Most may even prefer it over Richard Harris.
Dumbledore is my favourite character in the HP series though so maybe that's why it annoys me more.