There's good arguments for the case that gatherer communities actually had generally better health and far more free time than farmers and agrarian society.

Farming provided the calories necessary for a population that hunting and gathering could not support (so no going back) but required basically working all day to make it work and survive less than ideal conditions. But prior to farming people often had significant more free time.