He's a pretty easy sub-boss if you take him out with magic. He's a massive bastard to kill if you try anything else, though.

Get a Balder Shield. Upgrade it. Firstly, it's easily one of the best looking shields in the game, and secondly, it has excellent stability. With this shield in hand, I killed the Capra Demon without even using the stairs or hunting down the dogs first. Pretty sure the only one I killed was one that got in the way accidentally. xD Great shield.