> Yes, but why would anyone want to live on what is effectively a farm, but without the benefit of separation from other people or land (read: income) that a farm offers?

They don't want complete separation from other people. They want conveniences of the city/metropolis (access to jobs, entertainment, and education) while having a lot of space around their home for recreation & privacy.

They don't want farm work (required for that income) either - it's physically and emotionally hard and margins are thin and fragile depending on the weather.

FWIW, I grew up in Brooklyn, NY. I know about both urban and suburban living. I now live in Columbia, MD: a suburb from the late '60s planned with high ideals.

My three children each have their own room. They can ride their bicycles on our sleepy street without having to constantly worry about reckless drivers. They can explore the walking trails and wooded areas our community maintains. They can play baseball in one of the nearby fields without having to worry about breaking a car window in doing so.

My wife has a large crafting area. I have an office. We have a home theater. We have a workshop in our garage. We have a large sunroom which opens to a large deck suitable for entertaining many friends at once.

Getting even a fraction of what I described above in a city would cost a fortune.

> The whole reason for accepting being squeezed in tightly with other people

That's the thing, a lot of Americans really don't like this idea of being squeezed in tightly. People really don't like having a shared wall. And it's not that people like these tightly squeezed lot lines, they'd prefer 1 acre lots. These tiny lots are all they can afford while still being somewhat acceptable to them about commutes. Which Americans seem to really just not care about commute times when comparing to tradeoffs on house size and not having shared walls.

And in the end, you can only buy what the cities and towns allow to have built. Which is chosen by those who live there at the time. The cities then make single family structures a requirement, have minimum setbacks and lot sizes, have rigid separations between residential and commercial spaces, etc. So even those people who would want to own an apartment over a commercial suite in what you'd consider an urban area can't make that choice because that choice is illegal.

But people act like these zoning laws just come about on their own. The thing is, these zoning laws are popular. They get put into place because that's what the people who actually vote in local elections push for. I've seen proposal after proposal in cities around me to change zoning to allow density even in limited areas get fought tooth and nail by residents. I remember a project nearby where there was a proposal to build a mixture of 2-3 unit townhouses, some single-family narrow lots, and a tiny spot of commercial for like a coffee shop on land that was currently zoned industrial. All of this connected to the bike network, a large city park and a nature preserve nearby, and good transit connection at the end of the neighborhood. The neighborhoods around fought it tooth and nail and eventually the builder walked away after trying to negotiate for a few years. Well, the land was already zoned industrial, construction broke ground months later to build warehouses. Now instead of a nice neighborhood on my bike path there's warehouses with semi-trucks rolling through all day long. Good job, NIMBYs!

Because it's cheaper, if we're bring frank. The middle class is slowly being priced out of urban sprawls.