Everything you say is correct but it comes with a massive decrease in quality of life for the average person. Food will be 5x the cost, with less variety. Nobody wants to till the land anymore. Just like people 100 years ago didn't want to be hunters.
Are you willing to part with your smartphone and computer? I would bet not.
Why would I have to part with some of the cheapest things I own on a dollars-per-day basis? That's why GP mocked the comparison to smartphones.
That said, sustainable farming doesn't require very small farms, and it doesn't require more than 1% of the population to work on farms.