I'm wondering how much of it was racism, and how much was people simply being aware that the "worse" areas of town had lots of crime (which they associated with the color of the people living there, not knowing any better) and they didn't want that crime to come to their area.

If I'm walking a street at night and I see a guy with attributes associated with poverty, I'm more likely to switch to the other side of road than if he's wearing a suit, just saying.