There's only been three games I've ever bumped up to the harder difficulties: The World Ends With You (Ultimate), Dynasty Warriors 5 (Hard, haven't braved Chaos yet), and DMC: Devil May Cry (All difficulty settings).
Every other game I'm either scared of dying, because I really hate dying in video games, or playing it more for the story. Games like Mass Effect fall under both categories. For these, I play on either Easy or Normal, depending on what genre it is. I played the Mass Effect series on Easy, as I'm not a very good shooter.
If anything, games that don't give you the option to change difficulty annoy me the most. I love the Fable series, but I wish they implemented difficulty settings. Also games that punish you for not playing on the harder settings irritate me. Only unlocking trophies for playing on super mega death ultimate difficulty makes my completionist streak within me boil with anger.