LVT taxes only the land.
Some recent studies have shown that a properly implemented LVT avoids harming most homeowners, in fact you may come out ahead if other taxes are eliminated. (LVT alone is adequate to fund government spending and entitlements.) The homeowners who will typically feel a pinch are those who have inherited very large estates which don’t generate income. They may need to sell or develop part of their estate.
Land speculators, however, will be in serious trouble.