I've always had stubble. Even when I shaved it was there lurking, so i let it grow about 2mm and keep it there usually.

All men should have at least SOME facial hair.