It only annoys me when the phrase in question is kind of ambiguous when not endowed with said punctuation sign.
The lack of cookies is MUCH more distressing