Nonsense. The "weights" in "models" refer to probabilities.

Even the implicit claim that they could deterministically produce "the" correct answer with 100% certainty doesn't withstand any scrutiny.

Nevermind problems posed in English prose, complicated or philosophical questions. Is the correct answer to 2+2 four, or is it 1+3? When you you have 2 apples and give me one apple, how many apples do you have now; one, or half as many as before? What is the correct answer? Without a spaghetti of arbitrary axioms in the system prompt? Even if you come up with something clever about apples, it even fails at "when is your birthday". When it is today, should I say "today" or say the date? Not even God could decide that.

Arguably, the specifications for a compiler is also such a mess of axioms, and you can split hairs and say "it's all random anyway", but you'll still use a seatbelt instead of silly string, so what gives?

For compilers, give or take, there is a correct output for a given input (under which I'll include config, options, the targeted architecture, whatever). With LLM there is no such thing even if you do infinite mental backflips, and there won't be, because there can't be. Even if you could perfect the compilers that are needed to make the software that trains and drives LLM deterministic, you cannot make LLM fully deterministic without making them not an LLM.

If you can find a way to encode what a compiler would do to programs into the weights of a model so that produces the output of a compiler that would be a cool and completely useless feat, because it would probably be bigger, slower and impossible to reason about. But it would still be cool and I would still try it out.