I can't forgive Season 3 for stripping the most interesting elements out of the prison arc, softening and hopelessly muddling the Governor's character, and worst of all, reducing one of the source material's most enduring badasses in Andrea into a helpless waif whose every other line of dialogue seemed to be "THE GOVERNOR ISN'T REALLY A BAD PERSON YOU BIG MEANIES, LET HIM KILL A FEW MORE PEOPLE BEFORE YOU JUDGE HIM." Wasted potential irritates me more than no potential.
Season 4 was rock-solid, though, for the most part. As were the first few episodes of Season 5, when the threat was the Terminus Hunters, who really ought to have stuck around more for how interesting, sympathetic, and heavily set up they were. And then the show took off on a tangent to a random hospital full of indistinct jerks, and we've been in an only occasionally worthwhile tailspin ever since.