Rich people don't tend to have a sizeable portion of their worth tied up in their primary residence (and even then, IIRC there is a cap on capital gains exception), otherwise property tax would turn into a wealth tax for them which obviously they want to avoid. Non-primary residences still require paying capital gains. The inflated value you paid off with debt for a non-primary residence still gets captured as capital gain in the end when you actually want to sell the house for money.