alt Hacker NewsEveryone interested in LLM internals should read Sebastian. He's great.
The tldr here is that the recent "The Information" article[0] reporting GPT 6 Astra was using “recurrent depth” or “looped transformers" made it sound like it was some special new scary thing ("secret technique!") that made train-of-thought monitoring harder to do. In fact, it's just the same as stacking more transformer layers, except that you reuse the weights and so save GPU memory. It's still just producing one token at a time, and the token sequence positions aren't interacting in any "recurrent" way that's different from a regular LLM architecture.
So, you can still monitor train of thought with these models just fine... well, if you're OpenAI, anyway. Users haven't been able to see an unsummarized trace since o1 days, because the labs are worried about distillation of their models by Chinese labs.
(There are some legitimate interpretability concerns about stacking transformer layers endlessly, but we're known about that for a long time. And the "looping" here isn't really the source of any new issues here, except insofar as it's a cheap way to add more layers.)
[0] https://www.theinformation.com/articles/secret-technique-beh...
Replies
>made it sound like it was some special new scary thing that made train-of-thought monitoring harder to do.
It's not a "scary new thing" but ultimately no-one knows exactly how OpenAI have implemented looping. You might not be aware/remember but MoE transformers perennially underperfomed their dense counterparts until GPT-4. Similarly, making reinforcement learning really work with transformers wasn't figured out until o1.
And by Open AI's own admission, Astra's CoT is significantly harder to monitor and it exhibits a significantly greater control over its own CoT than any other model released.
Oh, is the principle of sparse universal transformers finally in SoTA LLMs?
I guess we did manage to eventually seriously crash into the wall "more compute than normal (non-looped/unique-weights) transformers can efficiently consume with the limited training data we have", plus massive focus on highly hands-off agentic tool use reasoning...
https://arxiv.org/abs/2310.07096
Edit: read much of the article, it's brute force predecessor was explicitly called out as an almost-ancient example:
> The looped transformer is nothing new, and the basic idea already appeared in the Universal Transformers paper from 2018
If the agent is able to “decide” when a loop should occur vs when an output token is produced, that effectively moves the CoT inside the architecture. While that’s not what is happening here, it’s clearly a plausible way we could see CoT disappear.
Schmidhuber must be rolling in his bed: https://arxiv.org/abs/2405.16039
It’s a little more complicated than that. While looped transformers can be unrolled a fixed number of times to save on memory, if loop depth is determined dynamically between tokens, a single transformer can compute any computable function between tokens.
To analogize, current transformers run a fixed-length program per step. Any program can be factored into a top-level loop with a fixed-length branching body (an interpreter). Dynamically looped transformers can run any program between tokens.
The safety argument for CoT monitoring is that in transformers information about the hidden state has to be communicated through the bottleneck of sampling a single token per forward pass. If not trained adversarially, it’s likely that a reasoning trace contains all the “bottlenecked information” we need to determine intent. But if we can compute arbitrary programs between tokens, the reasoning used is hidden.
It also opens the door to simple architectural extensions that would make the safety/monitoring side of things much more difficult.
It’s probably fine in practice at these scales though. If we keep each loop turn reasonable non-deep, we can probably recover most of the benefits by decoding “extended” CoTs from the residual stream at each loop turn between tokens. But that’s an area of active development.