How does that make someone selfish? I'm sure there are judgements you could make against someone who would prefer to be left alone, I just don't see how "selfish" could be one of them.

For me, one of the main motivations is suspicion of ulterior motives. If it really is just "hey I like your hat okay bye" that's one thing, and is generally harmless. But usually when someone approaches me they want something, either they're selling me something, or asking me to sign something. It's not that the initial comment is necessarily an issue, it's guarding against people pretending to have an innocent interaction as a foot-in-the-door technique.