I say it's because when men made the societal laws concerning boobage in public, they didn't admit that men's boobs turned them on. It was only correct to acknowledge that women's breasts were sex objects, and so they were stowed away beneath layers and layers of cloth.