> Err, no? That's not at all how llms work.

The Transformer architecture that almost all LLMs use are composed of many layers in sequence, each containing an attention component and a neural network component. The attention component copies data between tokens/vectors in the current context, and the neural network adjusts each individual vector in the context.

Notably, the neural network component behaves like a compressed index of the training data. Someone even made a blog post a year ago or so about training[0] a language model and then replacing the neural network with a traditional lookup of the training set. It performed nearly identically.

I mean, it's almost a tautology: we train models to repeat their training data, so obviously, it has to have an index of the training set inside of it. This index is heavily compressed, but compression is intelligence, and understanding that "the neural network is trained to learn patterns of text" and "the chatbot consults its training data" is nearly identical is a sign of intelligence.

> This all seems to dramatically underplay how interesting the actual attack was and what built up to it.

The interesting part is how the AI safety people, who have been worrying for decades about how AI superintelligence will kill us all to make one more paperclip than it could otherwise, failed to implement extremely basic IT security practice when dealing with potentially malicious software.

There is an additional conversation to be had about long-context time horizons but that's not relevant for this analysis.

[0] I am deliberately ignoring post-training but it does not impact this analysis. Post-training is already known to not substantially impart new capabilities onto models, it merely elicits what was already there. Effectively post-training is "re-weighting" the index of training set data.