alt Hacker NewsNon-determinism is configurable at the level of the mathematical model, but current production systems do not support deterministic evaluation of LLMs.
Non-determinism is configurable at the level of the mathematical model, but current production systems do not support deterministic evaluation of LLMs.
They do, though. Providers don't because batching makes it cheaper. Among the providers, DeepSeek seems to support it for v4 (and have actually optimized their kernels for batching), and Gemini Flash is "almost deterministic".