I always start on normal. I only change it if I'm getting my hand assed to me, or if there is no challenge at all. That's assuming the game lets me change it after I've started.

On second playthroughs, I almost always pick one difficulty level higher.