It doesn't bother me too much either, but I do find myself thinking like "there should be a Fire Flower in that thing" when I see a red block, for example.
It certainly doesn't affect the overall gameplay experience, it just makes the game feel just very slightly incomplete when they are empty like that.

I have considered making some sort of modest hack where I basically just take the original game and fill those blocks with Fire Flowers and Stars, I think that would feel quite satisfying actually.