I don't think we're approaching the limit of deterministic prompt -> source code mapping any time soon. Small variability in prompts produces medium variability in outputs. Building on previous outputs only extends the variability, where behavioral differences start appearing a few prompts down the line.

And the AI companies aren't focused on making this process any more deterministic, where repeated prompts are sampled using different seeds. I don't know if it's because the randomness makes the models explore a larger space and perform better, but it's probably intentionally done because it'd be quite easy to remove the stochasticity.

This doesn't match my experience with prompting AI at all. In every professional application of AI I've seen, there are a huge number of variables and inputs, over many turns, that produce the output. We see radically different results across different engineers using AI, and that variance is not 100% attributable to the prompt. I don't even think it's 50% attributable. See sibling comment for some delineation of these additional parameters.

The correct policy is what Linux adopted. "We don't care what tools you use, provide high-quality PRs."

Not really though, since the result of the prompt is not deterministic. It greatly depends on the model, the version, the harness, even time of day if the provider's infrastructure is currently overloaded and is silently degrading performance. Some things also require multi-turn interactions.

Except prompts aren’t deterministic, source code is.