In places like Detroit racism was very much the primary reason in the 1940s, 1950s, and 1960s at least. Some of it was just personal racism, pure and simple. See "blockbusting" techniques used by realtors who would hire a black mother to walk down the street with a baby carriage to give the impetus for white people to sell quickly and cheaply. Or a black family moving into a house, the white neighbors trying to push them out via lawsuit for breaking a racially restrictive covenant, then losing so that all the white people left the neighborhood (e.g. read about Orsel and Minnie McGhee in Detroit, I was the primary author of the National Register nomination for their house).

The racist pressure to move was also a structural issue supported by the Federal government. Redlining meant you couldn't get loans to buy houses in certain neighborhoods, and those neighborhoods were exactly where black people lived. Literal walls could be constructed between two similar communities of different races so that the white side could be higher rated and have better access to loans (e.g., the Birwood Wall in Detroit). A ton has been written about the topic, I can't really do it justice, but the idea that people just wanted a bigger house was not the case. A lot of Detroit actually consisted of single-family homes with yards anyhow, so it was literally white people leaving a house for a very similar one.