It also was built when Marley was still very much a massive empire. So it wasn't limited to the manpower inside the walls. This is clear from how many of it's actual construction materials were used.