He cared for his daughter, but there's no indication it was more than as a pet. He even literally kept her zombie corpse as a pet. He clearly has a weird corpse fetish thing going on. He supposedly cared for Andrea and then left her to be tortured/killed by a zombie. That doctor guy who was his assistant in town and supposedly knew him for a long time, he easily murdered that guy with his own hands from what I remember.

The story he told in the last episode was his brother took the fall for something the governor did and his dad beat the crap out of his brother, and the governer let it happen, and the moral of that story was his brother was weak. Consider the scene where he knocks on the door in the apartment building, that lady from his new family answers and he says "Goodbye" and turns to leave without a word, before she stops him. Consider the scene where the zombie is about to eat that new little girl in camp, and he shoots the zombie and then just turns and walks away, like no big deal. Consider the scene where he murders some guy for being "too good" and then calmly tells his brother how there is no right and wrong and if he doesn't agree with that, he's next to die. He uses people as tools. He lies without remorse. He kills people without even the slightest remorse. He doesn't show that he has any deep emotion other than instinct for self-preservation and anger and an inkling of attachment for his daughter. He forms connections with people on some level but it's not the same way normal people form connections. Can you imagine him abandoning his new family if "necessary"? I easily can.

Contrast with Rick, who does what he needs to survive but still feels really horrible about it for years afterward, who keeps questioning his actions, who is trying to keep his son from turning out like the governor. That more or less describes most or all of the main characters. They do horrible things but they're conflicted about it.

Granted everyone is living in a world that's set up nicely for psychopaths to prosper, because brutal things are necessary for survival. Granted I don't know what's going on in the governor's head (because he's imaginary), but the character seems psychopathic to me. I could be wrong.