Honestly my problems with him was more to do with the writing then the actor himself.
He just seemed constantly bitter and snarky, which was quite a contrast to the Banner portrayed in The Incredible Hulk movie. It didn't even feel like the same character anymore.
And that bothered me, because I loved his portrayal in The Incredible Hulk. This Banner I really had no reason to get behind him. I will say him and Stark had some fun bits together though.