Try calling him 'Dad'. Gafgarion's like "What Dad? You have a problem?"

I never made a whole run with him named 'Dad'.