alt Hacker News> LLMs are not deterministic
They are designed to be where temperature=0. Some hardware configurations are known defy that assumption, but when running on perfect hardware they most definitely are.
What you call compilers are also nondeterministic on 'faulty' hardware, so...
Replies
vlovich123 • today at 3:23 PM
Even with temperature and a batch size of 1 and fixed seed LLMs should be deterministic. Of course batch size of 1 is not economical.
troupo • today at 3:57 PM
with temperature=0 and no context. That is, a clean run with t=0, pk=0 etc. etc. will produce the same output for the same question. However if you ask the same question in the same session, output will be different.
To say the least, this is garbage compared to compilers
➕ show 1 reply
While they’re technically deterministic, they’re still chaotic, in the sense that changing irrelevant details in the input (such as writing “color” versus “colour”) can make the output completely different.