Pounding stone seems reasonable to me. Obviously I don't have any proof or even strong evidence but I saw a video that changed my perception of what is possible. It showed two old men making a millstone with hand tools: https://www.youtube.com/watch?v=lscs5ZgNQrE. The amount of labour involved and quality of the finished item was astonishing to me. Maybe you'll think that the hideous amount of labour needed to make a simple geometric shape makes you even more convinced the Inca has some other way to achieve their even harder task. But it is a fun video anyway.
Similarly astonishing to me is that Michelanglo's David was carved from a single piece of marble with a hammer and chisel. I mean, just look at it: https://en.wikipedia.org/wiki/David_(Michelangelo)
The video does not counter the parents argument about measuring fit.
What the masons in the video do is certainly impressive. Cutting organic shapes that fit perfectly together, as if they once were elastic, is another level.
Perhaps the did something similar to what dentists do when building on teeth so that the added material is not the only contract point when jaws are closed. That is, a contact sheet that leaves contact marks.