Serious injuries which will nag for the rest of your life do happen, but they're extremely rare. All things being equal, physical work is vastly better for your body than sitting at a desk for 8 hours a day. Desk work is associated with increased chances of heart disease, obesity, mental health issues, and much more. We're not meant to be sedentary, but desk work basically requires that.

The confounding factor is lifestyle. Kicking back after a hard day of physical work with a huge meal and a few beers feels amazing, but that's going to wreck somebody's health if that turns into a lifestyle. And similarly, a gym rat who works a desk job is going to probably be in excellent shape.