Judging by the color of one's skins is one of the easiest and oldest ways to discriminate. Only behind by the contents of one's genetalia. But the latter at least has a biological need to care for one another.

It's a bit of a chicken and egg situation. At least when someone looks at someone who "looks like them" but is poorer, they see "potential" (there's a name for this effect, but I forget). But the end goal is getting theirs first and foremost.