I agree. Most of the time, instrumental music is about atmosphere, and once it surrounds you with a certain feeling, your imagination is free to create its own story.
If the lyrics are bad and the music is bad (i.e. most popular music), I won't like it. If the lyrics are bad and the music is good (i.e. a lot of Radiohead's later material) I'll probably still really like it. But I don't think that even the most masterfully crafted lyrical tales can salvage bad music.