You seem to be ignoring the vast majority of human history before we developed farming. Agriculture societies are a relatively brief period of our collective history.
People moved from a hunter gatherer society to an agrarian society because the latter was easier.
But it also contain the most people. Industrial age contains even more people but it hasn't defeated agricultural age yet because it's still so recent.