Evrae, Sanctuary Keeper and Yunalesca were a real pain on my first playthrough. Never had much trouble with Seymour though.

They all go down easy once you've beaten the game once imo.