> it follows that it'll be able to replace all physical labor with robotics fairly quickly
This is not clear at all. Even the most expensive robotic/arms hands perfectly controlled fail to do thing as quickly and easily as human limbs, not to mention the chaotic environment that is the real world. It's clear that it can happen, but the timeline is not quick.
Just because a task is done with human hands, doesn't mean it has to be done with a robotic one. Our robot vacuum cleaners aren't Rosie pushing a standing vacuum.