They didn't select for pure physical fitness but they were already selected for fitness as a pilot and then again when they were selected from the pilots to train as an astonaut. Its not like they just picked arbitrarily from the potential pool of candidates and gambled on getting better than average.
Again, they don't select for pure fitness when it comes to pilots as well. The fact is that you didn't need to be super fit to handle those crafts. Fitness today is much more prioritized because astronauts spend exponentially longer time in space now than they did then and they have to work out in space to keep their bodies from getting used to being in space and zero gravity. They now spends months in space, previously max they would go for is like a week.
So no, pilots or astronauts are not "some of the most physically fittest people in America". They were exceptional human beings but lets be realistic.