If a minority has most of the wealth then the equilibrium supply may include a lot of supply of second homes, very large homes on large plots for the rich, properties sold at a premium based on how much they can extract from renters, and even investment properties occupied by nobody whilst still having insufficient small basic homes and dense housing.
Capital that could be invested in better serving the bottom half has to compete not only with the use of those resources to further enrich the rich but other investment opportunities.
There's a couple of "ifs" there and the scenario seems implausible. If I look at the prime real estate in a city it tends to be a lot of skyscrapers rather than very large homes (with occasional exceptions like say a Buckingham Palace). But it looks like the economic equilibrium is lots of cheaper apartments rather than large homes for rich people.
> ... and even investment properties occupied by nobody ...
Not much of an investment. Something is wrong if that is happening, probably manifesting as a lack of supply. Otherwise what is the point of an "asset" that doesn't generate income, degrades over time and could easily be rented out at a profit rather than sitting unused?
Whatever scenario there is where it makes sense to have an empty property, assuming a sane policy backdrop, it'd always be better for the owner do what they were going to do anyway but also rent it out.
If you want second homes to be used productively, just allow folks to list them on AirBnb for planned short-term rental. Long-term rentals are a total non-starter politically for second homes, since you obviously can't ethically throw out someone who regards that rental space as their home. But short-term is actually fine.
As for larger homes, people should be allowed to live in there as larger, extended family groups - a common pattern in non-Anglo cultures. Ban "single family" restrictions since they amount to unconscionable discrimination against such reasonable living arrangements.
There is more than enough land for everyone, and rich people aren't really competing for the kind of housing that poor people are competing for, e.g. smaller plots with smaller homes. The demand of the rich does not eliminate demand of the poor, so the market produces different kinds of housing for different clientele.
Think about it this way: assume you supply all the housing to all the rich people. Then there still remains untapped demand of others that can be fulfilled by further production of homes for those specific people.
This story fails when land becomes restricted, which is exactly what zoning laws cause. Zoning is a big harm to the poor.