alt Hacker NewsYes, but at least now we're comparing artificial to real neural networks, so the way it works at least has a chance of being similar.
I do think that a transformer, a somewhat generic hierarchical/parallel predictive architecture, learning from prediction failure, has to be at least somewhat similar to how we learn language, as opposed to a specialized Chompyskan "language organ".
The main difference is perhaps that the LLM is only predicting based on the preceding sequence, while our brain is driving language generation by a combination of sequence prediction and the thoughts being expressed. You can think of the thoughts being a bias to the language generation process, a bit like language being a bias to a diffusion based image generator.
What would be cool would be if we could to some "mechanistic interpretability" work on the brain's language generation circuits, and perhaps discover something similar to induction heads.
Replies
> comparing artificial to real neural networks
I had a sad day in college when I thought I'd build my own ANN using C++.
First thing I did was create a "Neuron" class, to mimic the idea of a human neuron.
Second thing I did was realize that ANNs are actually just Weiner filters with a sigmoid on top. The base unit is not a "neuron".
>Yes, but at least now we're comparing artificial to real neural networks
Given that the only similarity between the two of is just the "network" structure I'd say that point is pretty weak. The name "artificial neural network" it's just an historical artifact and an abstraction totally disconnected from the real thing.
> Yes, but at least now we're comparing artificial to real neural networks, so the way it works at least has a chance of being similar.
Indeed, and I wasn't even saying it's wrong, it may be pretty close.
> What would be cool would be if we could to some "mechanistic interpretability" work on the brain's language generation circuits, and perhaps discover something similar to induction heads.
Yeah, I wouldn't be surprised. And maybe the more we find out about the brain, it could lead to some new insights about how to improve AI. So we'd sort of converge from both sides.