What's in need of explanation? Computers are designed to be deterministic, and LLMs run on computers, so LLMs must also be deterministic.

There caveat to that is where external inputs give the illusion of non-determinism. Thread execution order, like we already discussed, is one such example. Technically still deterministic if you understand the external inputs, but for the sake of discussion we can consider external inputs to be non-deterministic. Which is why compilers usually end up being non-deterministic by default. However, computers are designed to be deterministic so there are ways to avoid introducing those external inputs, albeit often at the cost of performance. LLMs and compilers alike can be run deterministically.

LLMs have one additional property not typically found in a traditional compiler — a call to rand() — but rand() is also deterministic when configured with a constant seed and can also be turned off completely by setting temperature to 0.