"…not because they are racist/prejudiced or whatever…"
Perhaps wealthy actors (or those representing them) are attempting to distract the aggrieved—focus their anger toward some non-wealthy group.
This is seriously the basis of the manual of populism as mass control tool: identify something attacking their group identity, hang it before their eyes, and they will follow you blindly in every other aspect. Many books have been written about this and nowadays we can see their implementation at work about everywhere.
You can spin many things to sound as racism if you want them to.
My small eu country has in the recent years started importing workers from asian countries, and people are against that... of course it's branded as 'racism' in the media.
The reality is, that some businesses have been having troubles with finding workers to do hard work for minimum wage (which is easily solvable by paying people more and creating better working conditions, but hey, think of the profits), and importing foreign workers means that wages can stay down. We're talking about AI and how we'll all lose jobs,and at the same time, importing new people, and that doesn't compute for many, especially those underpaid who can't negotiate for better options because the government approved new foreign workers in their field.
Then you have the more on-topic problem of importing thousands of "outsiders" into a relatively small city (~300k pop), without building any housing, neither for them, nor for locals, who want to move out their 30+yo "kids" out of their childhood bedrooms, nor for others living a bit further away who'd like to live closer to work. Sure, you can fit 8 "poor foreigners" into a 2-bedroom apartment with 4 bunk beds for 200/month each, and it'll be cheap for them, but for a young couple wating a child, 1600eur for such an apartment is around 1/2 of their combined income, so nope, no apartment and no kids for them, since they have nowehre to put them, and "we need foreigners, because our people are not having enough kids".
Then the third problem is integration... and the government does nothing about it, no language courses, no nothing... the laws say that businesses must use our native language, but in some areas, you can interact even with waiters or shop clercs (so, first line of interaction with customers) who can't speak a word of it.
But hey, it's easier to say "racist people hate foreigners", than to solve the issues.
This seems like an interesting theory, but when I look at the two ways it may be actuated it seems to fall apart.
Either there is explicit collusion where they meet and discuss how to do these things, or there is implicit collusion where they all act in their own best interests and the results is the same. Neither of these mechanism seems particularly likely to me.
I would be happy to hear some thoughtful explanation of how other people approach this issue