I believe the reasons we "regressed" into agriculture from hunting and gathering are much more complicated than "we moved into more marginal land".

It does appear that the median hunter gatherer life was better than the median farmer life. But I'd wager that to be true in most areas.