Most hobos I've seen are druggies, so if I gave them money, it'd go to drugs, not food. Furthermore, there's organized begging around here, so it's hard to tell who actually needs the money.
I usually just walk past them, and no one has cursed at me for doing so. I guess they're just used to people passing by them anyway. However, if one of them would actually swear at me for not giving anything, I would definitely get pissed off at him.