This is probably about extremes being bad. Having an extremely sedentary lifestyle is bad, but also having an extremely strenuous one is bad too.
I used to lift weights regularly. I'd go to the gym three times a week for an hour or two at a time. I'm pretty strong naturally and thought my training was going quite well being able to bench 1.5x my bodyweight, squat and deadlift more than 2x etc.
Then I paid some guys to move house for me. Actually, my job paid, I have still yet to pay for this service myself. They were lifting whole chests of drawers without even emptying them. It was crazy. I've since done plenty of this work myself (moved house three times by myself), but I do take the drawers out etc. Basically I work more intelligently and take more time.
What the moving guys were doing is harder, less safe, and they are doing this day in, day out. Add to that poor diets (both seemed to be fuelled on crisps, Coke and fags) and the differences become more clear.
So, like with anything, don't be too extreme. Too much heavy lifting will be just as bad for you as too little.