alt Hacker News> But what if our neurobiological reality includes a system that behaves something like an LLM?
With every technological breakthrough we always posit that the brain has to work like the newly discovered thing. At various times brains were hydraulic, mechanical, electrical, like a computer, like a network. Now, of course, the brain has to be like an LLM.
Replies
All of those analogies were useful in some ways, and LLMs are too.
There's also a progression in your sequence. There were rudimentary mechanical calculating devices, then electrical devices begat electrical computers, and LLMs are a particular program running on a computer. So in a way the analogies are becoming more refined as we develop systems more and more capable of mimicking human capabilities.
Yes, but at least now we're comparing artificial to real neural networks, so the way it works at least has a chance of being similar.
I do think that a transformer, a somewhat generic hierarchical/parallel predictive architecture, learning from prediction failure, has to be at least somewhat similar to how we learn language, as opposed to a specialized Chompyskan "language organ".
The main difference is perhaps that the LLM is only predicting based on the preceding sequence, while our brain is driving language generation by a combination of sequence prediction and the thoughts being expressed. You can think of the thoughts being a bias to the language generation process, a bit like language being a bias to a diffusion based image generator.
What would be cool would be if we could to some "mechanistic interpretability" work on the brain's language generation circuits, and perhaps discover something similar to induction heads.