I think part of the reason is that the older games had nothing to give you except a challenge. There were no fancy graphics, no movie-like cutscenes, and no realistic mechanics. The only thing those games had to offer was gameplay, and one of the only things available in the gameplay department was to make it challenging. I am sometimes amazed at the patience I had as an 8-year-old to play some of those older games. We put up with some absurd levels of difficulty made even more impossible by frequently awful controls, because that is what games were back then. Sometimes I do miss that sort of challenge, but I recognize that I probably wouldn't have the patience to retry the same level 500 times in a row like I used to.

Nowadays, developers have so many options, and they emphasize different things, such as graphics. And they're trying constantly to appeal to new gamers, which involves making things easier (or at least having the option to make things easier), instead of more challenging.