> AI is not deterministic, it will be making tons of mistakes.

Just set the sampling temperature to zero and remove any unintended non-determinism during the parallel computation of the token probability distribution. The problem is solved? Of course, not. Non-determinism has little to do with LLMs' mistakes.