> And human developers are also non-deterministic. There is always the nondeterminism even before AI.

Humans are not machines. The entire point of any technology (including specifications derived from consensus, such as the metric system, democracy, etc) is that it operates deterministically in the service of (as opposed to outside of) actual people with actual needs. This is what makes a tool a tool, and not a human. To me, it all sort of boils down to the shortest path between an intention and it's result. LLM generated code can massively shorten the time between A and B, but not necessarily shorten the path.

The feature that makes LLMs great and interesting is the same feature that makes them hallucinate. This is embedded in the entire idea of coherence being the byproduct of nudging weights in proximity as opposed to being constructed toward an unknown future. I thought it could be cool too, but a deterministic AI (zero temperature, frozen model, or whatever..) would _only_ really serve to be a leaky abstraction over its training sources. Which maybe says more about our access to information and desire to understand it.

I don't think there's a clear cut line (or even value in one!) between LLM output being good or not, but without recognizing that LLM output will always be riffing on prior inputs as opposed to being intentionally constructed toward an unknown future, we've probably lost the plot.