>I wonder if the hunter gather societies could have grown larger if they put in the same level of work as the agricultural societies?

I think it is about organization and population density. A hunter gatherer society is not going to be able to field an army of tens of thousands of people, as an agricultural society can. Hunter gatherers are also limited in their technology by their continual movement.

The Mongols were a nomadic society and very successful militarily (for a while). But they kept large numbers of animals and weren't hunter gatherers.