alt Hacker NewsI understand linear algebra. It seems like he probably does, too.
Are you sure you have read and understood his point?
It seem quite possible that the flattened-model-weights ingested into a model have a much greater fidelity of representation in a similar model's KV cache than they do in a human's working memory, and that being able to loop and digest and chunk makes it unsurprising that models can excel at this type of task.
I think that's the point the person is making. Do you disagree? If so, why?
Weights are not sequences of tokens in any sense. This is pure nonsense. We don't "see them" [presumably, the weights] as vectors (they are matrices or tensors).
There is nothing obvious or expected about weights being easily explainable or being enough to explain, because the weights are static: you need to look at the inputs, the activations, and the weights all together, you can never understand a model with the weights alone, unless it is a trivial model. The person isn't making any coherent points at all.
There are also deep reasons to expect hard limits in intrepretability in general: https://www.seangoedecke.com/alignment-is-impossible/#why-me....
EDIT:
> It seem quite possible that the flattened-model-weights ingested into a model have a much greater fidelity of representation in a similar model's KV cache than they do in a human's working memory,
Bringing up the KV cache is a sign you don't understand what you are talking about here. The KV cache is a computational efficiency thing and has no impact on how these models actually work (and isn't needed at all). The vague analogy to human working memory also just takes the cake here.
> and that being able to loop and digest and chunk makes it unsurprising that models can excel at this type of task.
This is how it does... all tasks.