You don't sound like an asshole because you aren't saying bad things about the people who have the least.

It's up to you, you can just, to yourself, write down the words you'll use to write down that you'll ignore them to become more comfortable with your boundaries.

Personally I don't think handing out cash is helpful so it's not about charity it's my advice to pre-plan how you'll respond to be more comfortable than you are in reaction to these situations.