alt Hacker NewsEhm, no! The same prompt does not always lead to the same output.
Have you ever tried to achieve consistently deterministic output from an LLM? I have, and it's not easy.
That means output differs between machines and architectures. Running inference on CPU vs GPU also affects output. Even running the same prompt twice in a row on the same machine can lead to different outputs because a prompt that was partially stored in the kv cache will result in different output than an uncached prompt.
LLM output is very much not deterministic!
Replies
These are all implementation details.
At the end of the day, an LLM is just a very big mathematical function. That is, by definition, deterministic. A particular implementation might give up on determinism for the sake of higher efficiency, but it you want a deterministic LLM, it can absolutely be done.
It's not deterministic due to implementation details in floating point numbers and order of execution.
If you ran an LLM with infinite precision and guaranteed order of execution, it would be deterministic.
(I think determinism is overrated. Being deterministic does not make LLMs more reliable or correct.)