Taxing primary residence or the land its on will destroy middle and lower class and allow rich ones to go even further hand.
Luxury tax and tax on secondary homes would probably be better.
Have you done the math? Under a revenue-neutral shift that taxes buildings less and land more, the median homeowner comes out ahead:
https://landeconomics.org/reports/spokane-report
I think what you're missing is that land in the wealthiest parts of town is worth exponentially more than land out in the suburbs
LVT taxes only the land.
Some recent studies have shown that a properly implemented LVT avoids harming most homeowners, in fact you may come out ahead if other taxes are eliminated. (LVT alone is adequate to fund government spending and entitlements.) The homeowners who will typically feel a pinch are those who have inherited very large estates which don’t generate income. They may need to sell or develop part of their estate.
Land speculators, however, will be in serious trouble.