I don't think anyone "only" plays games for plot, they merely play games that focus on the plot and have more to them than that. Otherwise they'd watch a movie or a TV series.

For everything that most of the games you mentioned offer, they don't focus on the story while still offering good gameplay, they focus on the gameplay while still offering good story. However, for the most part, these plots aren't the kind of plots that would work for a Final Fantasy game. JRPGs tend to be a lot more global in their premise, and involve a lot more fantasy. Zombie plots and nuclear war plots are great and all, but they do not belong in a Final Fantasy game.

The only problem with JRPGs from my perspective is that the plots aren't as good as they used to be. The gameplay, for me, is still decent. It could definitely improve, don't get me wrong - but if these JRPGs had incredible plots then it wouldn't matter so much.

I can also see a lot of reason in Skyblade's thought regarding plot vs. characters.

I'm simply trying to get to the core of why we play these games, especially since all we ever seem to do is whine about how they are not as good as the old days but still we stand in line for the next installment because we've apparently been trained to. So why do we keep doing this to ourselves?
Every individual is different. I personally always want to play the next FF game because I know it will be something different. The previous game can be bad and there's still every chance I'll like the next one. That's the beauty of a Final Fantasy game for me in particular - each game is different in ways that shooters can't manage. I like multiple genres, and that's because I enjoy variety in my video games. RPGs happen to offer the largest amount of variety within their own genre, in my experience. JRPGs tend to focus more on the story, and I like a good character and a good "global plot", so these things work for me. I also like the surrealism and fantasy involved in them - they take you to a new world each time. That's awesome.

Is there some inherent value to the genre that is understood but not said that keeps us coming back or are we simply Pavlov's Dogs salivating to a bell that signals the JRPG of yore we grew up on, even though its been years since the genre offered an experience on par with it?
It seems to me you're talking more about yourself than about other people. I, for one, enjoy the games I play. You should try that sometime... focus on just having fun instead of analysing everything.

I don't want a complete lack of challenge in my JRPG. I don't want it to be TOO challenging, either. Personally I am happy with just having to have fun. What you define as a complete lack of challenge is different to what others will define, though. You probably are much better at overcoming challenges that other people might have more trouble with. This is A Thing That Happens. I'm better at gaming than some people, I'm worse than others. Same goes for you. If you happen to prefer your games ultrahard, then good for you, but that probably won't fly for most people.

Gameplay should never be confused with difficulty. The two are linked in some ways, but you can have games with incredible gameplay and next to no difficulty and vice versa.