I don't think that is necessarily true. I believe there are, for instance, many reasonably wealthy people who live somewhere outside California and would not move into a dumpy old apartment in LA or SF but might move into a fancy new one. In other words, they will not occupy an additional housing unit (in a given market) unless it is "nice" enough.
I am in favor of building new housing, but I'm even more in favor of reducing wealth inequality. I think we can do both, but we need to be deliberate about it.
Taken to extremes you'll have all wealthy people living in California, and the rest of the country available to us plebs.
The greatest tool we have to reduce wealth inequality is make it so people can buy homes - and the biggest levers we seem to have there are making supply available in general, and making jobs available where there's already supply.
>I am in favor of building new housing, but I'm even more in favor of reducing wealth inequality. I think we can do both, but we need to be deliberate about it.
... And that's why nothing gets done.