Real property is a bundle of rights in land and its fixtures – that's the standard definition in academic legal theory
If there was already a zoning or restrictive covenant, and you knew about that when you bought the land – then the ability to do with the land what that zoning or covenant prohibits is not part of the rights to the land you acquired, and the enforcement of that zoning/covenant isn't denying you any of the rights you paid for.
Besides that, private property is (almost always) a creation of the government. [0] I own a house. If I trace the chain of land titles, it ultimately ends in a government land grant; the government got it by stealing it from the Indigenous people. The government land grant was issued on the condition of obeying the government's laws as in force from time-to-time. You can't sell what you don't own; the original land grant recipient didn't own the land independent of that condition, so none of the subsequent purchasers could either. So the government restricting my ability to use the land I purchased isn't an infringement of any of the rights I actually paid for.
[0] I live in Australia; under Australian law, "native title" is considered to pre-exist Australia's current British-imposed system of governance, but all other land titles derive their legal existence from being created by that system; I don't think the situation in the US is fundamentally different, all non-Indigenous US land titles derive from action of the state or federal governments, or the colonial governments, or the Mexican or Spanish or French or Dutch or Russian or Hawaiian governments
Remember that there are plenty of houses today in America which were bought under the explicit promise that their neighbors would always be white people. That got taken away. Why should the promise of single family housing be any more precious?