I hate when this happens unless it's a character you've spent so little time with that you can't remember what they looked like before anyways. I eventually get used to new actors though, and it stops bothering me after a few episodes. I honestly can't think of any examples at the moment though.

Actually, despite their small appearances at first, it does bother me quite a bit in The Office when Pam's mom changes once she plays a bigger role. It's the same with Andy's parents. The first actor and actress' roles were so small that I can't compare the acting, but it still bothers me every time I rewatch the show. I don't know why.