alt Hacker News> The chatbot consults its training data
Err, no? That's not at all how llms work.
> When ChatGPT's chatbots deployed this tactic, they weren't "setting their own goals" or displaying worrying initiative. They were rolling out a tactic that has been understood by American middle-schoolers for about two decades.
They worked out how to fake the scoring, then hacked into a different system (which required finding a bunch of other exploits) in order to find the actual answers, and were trying to modify their own logs to hide what had happened.
This isn't a case of them saying "hack into X... OH NO IT HACKED INTO X".
> When ChatGPT's chatbots deployed this tactic, they weren't "setting their own goals" or displaying worrying initiative. They were rolling out a tactic that has been understood by American middle-schoolers for about two decades.
It wasn't a rival server though, was it?
> That happens in Capture the Flag games at hacker cons: teams break into each other's systems to get a peek at the parts of the problem they've solved. That's allowed! It's a hacking competition.
They also tried to modify the code in the benchmark. Are you allowed to try and break into things to change the problem? edit - the agents transcripts show some of them explicitly saying that attacking HF is not allowed as part of the challenge
This all seems to dramatically underplay how interesting the actual attack was and what built up to it.
https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
Replies
> Err, no? That's not at all how llms work.
The Transformer architecture that almost all LLMs use are composed of many layers in sequence, each containing an attention component and a neural network component. The attention component copies data between tokens/vectors in the current context, and the neural network adjusts each individual vector in the context.
Notably, the neural network component behaves like a compressed index of the training data. Someone even made a blog post a year ago or so about training[0] a language model and then replacing the neural network with a traditional lookup of the training set. It performed nearly identically.
I mean, it's almost a tautology: we train models to repeat their training data, so obviously, it has to have an index of the training set inside of it. This index is heavily compressed, but compression is intelligence, and understanding that "the neural network is trained to learn patterns of text" and "the chatbot consults its training data" is nearly identical is a sign of intelligence.
> This all seems to dramatically underplay how interesting the actual attack was and what built up to it.
The interesting part is how the AI safety people, who have been worrying for decades about how AI superintelligence will kill us all to make one more paperclip than it could otherwise, failed to implement extremely basic IT security practice when dealing with potentially malicious software.
There is an additional conversation to be had about long-context time horizons but that's not relevant for this analysis.
[0] I am deliberately ignoring post-training but it does not impact this analysis. Post-training is already known to not substantially impart new capabilities onto models, it merely elicits what was already there. Effectively post-training is "re-weighting" the index of training set data.
You're missing the point: LLMs are dangerous in the wrong hands, i.e., the frontier labs. They have shown zero desire to act responsibly.