It all depends really, for instance some people like Robbie Williams and Girls Aloud irritate me so much I can't even judge their music objectively because of it. Also Oasis, who have some great songs, but the hideous Neanderthals that are the Gallagher brothers almost ruin it for me. But on the other hand though someone like Damon Albarn is a pompous idiot at times, he's still a brilliant musician with Blur and Gorillaz in my eyes. Also Morrissey has given some repellent views in my opinion on people who do research on animals, but I love The Smiths still.
So it really depends how irritating I find the person/band in question.