This is actually very difficult for a significant number of people. Some people really struggle with saying "no" or enforcing boundaries, some people are very wary of negative interactions with strangers. If you are relying on people to explicitly push back on you, rather than reading more subtle queues, you are quite likely adding stress to someone's day.