alt Hacker NewsLLMs are not deterministic per my understanding. A program always produces the same output for the same input and instructions (ignore FP accuracy for now). How is determinism achieved here?
Replies
armchairhacker • today at 1:10 PM
LLMs may be deterministic for a subset of inputs, if one output (or intermediate layer) neuron-state probability is significantly higher than the rest. My understanding is, when probabilities are close they diverge.
LLMs produce a distribution of token probabilities which is then sampled. This sampling is the only random part of the system.
If you just take the most probable token every time, the system becomes fully deterministic. We don't do this as the output becomes more stiff and less creative.