The people of the United States are broadly free to build a home wherever they can afford to, comrade, including on land that would otherwise be used for farming.
(Actual answer: I know a bunch of people who live in houses in the middle of seemingly-nowhere in rural Ohio, and almost none of them farm anything at all. They just seem to like the space and the quiet and the desolate isolation.
The only farmer who I know is my parents' neighbor, who has a house few miles away from their place.)
I like your version of America. Sadly, California's not that free. Some billionaire can't just buy up some land and just put in apartment/office/factory tower as they please, the local government and residents just aren't going to stand for that.
Depends where you live. In my state you pretty much cannot build any kind of residence on land that is zoned for agriculture.