> rich people aren't really competing for the kind of housing that poor people are competing for, e.g. smaller plots with smaller homes.

This disregards basic geometry. Sure, in some rare situations you only have one small plot of land surrounded by existing construction or natural boundaries. But, in the majority of cases, you have one large plot of land, and you can either construct one big house on it, 5 smaller houses, 10 small houses, or 200 apartments in a block. The rich are absolutely competing for this lot with the poor.

And as inequality goes up, the rich can even start contemplating buying up surrounding properties, tearing down construction, and transforming a small plot into a much larger one.