I think you’re missing the point. It’s not you gaining more freedom with the land that’s the problem. It’s that all your neighbors now have the same freedom.

I bought my house with a close neighbor because I know the neighbor can’t build a steel mill in their backyard. If the government comes in and says we’re all free now to build steel mills, that gain in freedom is a loss in value for me.