I’m shocked how many otherwise well-informed people don’t understand or agree with this very fundamental fact of just how little we actually understand about why LLMs work as well as they do. They figure “it’s science, of course there’s math and theory behind it.”

AI research is almost as purely empirical as the gradient descent loops its practitioners use to optimize their models. “Why” anything at all works is barely an afterthought.

>But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.

We might not understand particular "emergent" capabilities, but the low level mechanism is not just understood, but a deterministic algorithm with a handful of basic componets, that are well understood themselves.

Please go on, else you risk sounding like the person you’re criticising. The structure of the neural network is somewhat opaque because it’s hard to understand as the individual weights can’t be usefully interrogated, and naturally, it comes from big datasets which a human brain can’t really absorb in toto. Your comment was interesting so I’d like more of it.

They don't "make decisions".

That's like saying "my d20 decided to roll a 17"

>> an LLM is completely opaque

And despite that, although they are not like that in practice as there are too many uncontrolled variables, with temperature at zero, for the same input they produce always the same reply.

We understand how networks compute decisions, though explaining every internal influence remains difficult.

thats no different from not understanding why a sufficiently complex and obfuscated binary of a program "makes decisions"

Isn’t it convenient that nobody understands? How could we possibly regulate something that isn’t understood? It’s like social media all over again. We can’t be responsible for someone else’s content; it’s not us so you can’t penalize us!!