I mean, why does it ruin your day? It's just some random person - you'll likely never see them again, or you'll know to avoid them in the future. Why is the opinion of some rando weighing on you so much?
have you had actually negative interactions like that? they sting for years, even after hundreds of mild-to-positive ones. the brain focuses on risk-minimization and not reward-maximization.
This whole thread is about wanting to talk with strangers because it makes you feel good, if approval from strangers makes you feel good the natural corollary is rejection from strangers can also make you feel bad. It would be bit weird to go out of your way to talk to people because you'd enjoy their kindness but then when they're unkind turn around all like "oh I never cared about you anyway". Isn't it?