If you can hold up a sign, begging for money for hours on end, you can work (and get benefits).
Also, I've been scammed out of too much money over the years. The money I give up is only for the mutual exchange of goods and or services. While giving a 'homeless' person some money may make you feel good, it is not very utilitarian. If selfless acts are what I'm after, there are other ways to help out such as with awesome charities, volunteer work, anonymous stuff, etc.
Another city did an experiment: they built and maintained houses for the homeless and almost eliminated homelessness. I'd gladly pay more taxes for a program like that. 🙂
Not giving a homeless or supposedly homeless person money is one thing. Personally I've given homeless money, I've bought a few food, etc... Some of them are genuinely in trouble, but some make hundreds of dollars a day panhandling and homeless is their job. One homeless white guy near my house is the nicest dude ever and he gets paid a few dollars here and there and food to clean up the two or three gas stations on this one stretch of road. He's genuinely homeless. Another guy I saw begging one day, and then I came back by a few hours later and saw him hop into his car. So there's plenty of apprehension when it comes to financially aiding a homeless person. I remember once my dad tried to buy this guy some food and he said he'd prefer cash. So stuff like that is understandable.
Now if you see someone hurt or ill and you do nothing, you're just a bad person.
Apathy. There's almost 7000 homeless in San Francisco. I used to give people money, but after being haggled for cash by sometimes dozens of people a day, for years, I barely even notice them when I walk by now. They've more or less blended into the scenery.
Even on a conscious level I'm not inclined to give people money, though. Most homeless are either drug addicts or mentally ill, in which case giving them money is pointless (oftentimes they won't even buy food with it, just drugs or booze). As DDM said, their are more constructive institutions to give money to if you were so inclined to help others, like the Red Cross, etc.
I think a lot of people assume that they are homeless because of a drug/alcohol problem, and if you give them money they will simply spend it on drugs/alcohol.
A lot of other people think they are just lazy.
It's unfortunate because a lot of people are homeless not because of those reasons but because of mental illness or something of the like, but they get lumped into the other groups simply through proxy.