Back in the day, games were very limited in the variety of content or emotional stimulation that they could provide through gameplay, visuals, story, etc. To compensate for that and ensure their games would get a certain number of hours, developers made them difficult. Another contributing factor was that controls were not as tight or precise as they can be today and that lended itself to increased difficulty.

Modern games have a whole lot more options than they used to and as such don't need to rely on difficulty to create a memorable experience. The focus now is on the overall gameplay experience and not as much on it being 'hard', and difficulty is now used as an option for those who want it or as an intended draw of the particular game.

For me, there are so many games I want to play that I'm not really interested in spending a whole lot of time trying to figure out a particularly hard game. I like a challenge in the sense that I want to be able to use my brain to get through it, but nothing to the extremes of older games that may require you to attempt a particular section many times to complete it. I also like constant saving in games so that you aren't repeating elements you've already mastered just to get to that one difficult part. Man that pissed me off so much in old games.