alt Hacker News> As long as AIs are either non-deterministic or chaotic
You just hit the nail on the head.
LLM's are stochastic. We want deterministic code. The way you do that is with is by bolting on deterministic linting, unit tests, AST pattern checks, etc. You can transform it into a deterministic system by validating and constraining output.
One day we will look back on the days before we validated output the same way we now look at ancient code that didn't validate input.
None of those things make it deterministic though. And they certainly don’t make it non-chaotic.
You can have all the validation, linters, and unit tests you want and a one word change to your prompt will produce a program that is 90%+ different.
You could theoretically test every single possible thing that an outside observer could observe, and the code being different wouldn’t matter, but then your tests would be 100x longer than the code.