Depends on the type of game. If it's a sports game like FIFA or a roguelike such as FTL or Rogue Legacy I want a challenge. It means nothing if I lose, no progress is lost and the next game will be different. If it's too easy then I get bored and move on.

If it's a sandbox game I just want to dick about and cause mayhem and making it harder to do that isn't a good thing. If it's a story heavy game and I'm stuck in one place forced to repeat the same checkpoint over and over again that's not so much fun either.

Also it depends how the higher difficulty is implemented. Adding more health to enemies and having the player be taken out by a stiff breeze adds difficulty, but does it add fun? Not for me. I can sit behind cover plinking away at an enemy's health for 10 minutes until it drops if I want to, sure. That's more of a test of my patience than a test of my skill though.