GPT-6 Astra, looped transformers, and hidden reasoning

For the research focused, there are some references in my blogpost here on what kinds of computational problems minimally require how much CoT to solve: https://blog.wtf.sg/posts/2023-02-03-the-new-xor-problem/

Notably Will Merrill's work: https://arxiv.org/abs/2310.07923

As for how universal transformers (looping transformers, but everyone has since forgotten prior work) will affect this, Will Merrill (again) has a paper here (https://arxiv.org/abs/2503.03961) that discusses exactly this.

The original universal transformers is called "universal" because if you allow for per-token looping decisions, it can theoretically be Turing complete without needing CoT (some nuance here about levels of precision used).

As for whether having little or no CoT is "unsafe": It isn't clear that the model's CoT reveal how they actually arrive at the answer. As an example, what if they provide an answer before the CoT? (https://arxiv.org/html/2603.01437v2) If this is already in question, we shouldn't be relying on the CoT for monitoring the model's reasoning.

As always there is a lot of nuance to the topic once you get your hands dirty with the details.

Astra was insane until Monday but something happened on tuesday, now it feels like Sol. I grieve for the lost productivity but i hope they may give us the original Astra back.

> I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4. OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it’s a core goal of our current research program.

- Jakub Pachocki (OpenAI’s Chief Scientist)

I wonder how helpful this actually is for alignment? Didn't we already determine that they know when they're being evaluated, and they just say what they think you want to hear?

If you loop an entire transformer model on itself, that seems like by-definition hidden reasoning.

If the output of the model is its reasoning trace, and you simply feed that back into the model again at inference time instead of outputting it - then it is by definition hidden (but I would expect you could pull both this trace and a further-down final output trace out)

The MSPAINT computer use demo made my jaw drop.

I guess it's not too different from the SVG pelicans, in terms of what it's doing, but it's still amazing to see it working in real-time like that.

> So, the whole idea here is that we increase the effective depth from 22 to 44 block applications without adding another set of transformer weights.

From what I gathered, LLM inference is bottlenecked on memory, right? Which implies there's "spare" compute we haven't been using? Does reusing the weights like this allow us to utilize it? (Do more math per unit of memory?)

Everyone interested in LLM internals should read Sebastian. He's great.

The tldr here is that the recent "The Information" article[0] reporting GPT 6 Astra was using “recurrent depth” or “looped transformers" made it sound like it was some special new scary thing ("secret technique!") that made train-of-thought monitoring harder to do. In fact, it's just the same as stacking more transformer layers, except that you reuse the weights and so save GPU memory. It's still just producing one token at a time, and the token sequence positions aren't interacting in any "recurrent" way that's different from a regular LLM architecture.

So, you can still monitor train of thought with these models just fine... well, if you're OpenAI, anyway. Users haven't been able to see an unsummarized trace since o1 days, because the labs are worried about distillation of their models by Chinese labs.

(There are some legitimate interpretability concerns about stacking transformer layers endlessly, but we're known about that for a long time. And the "looping" here isn't really the source of any new issues here, except insofar as it's a cheap way to add more layers.)

[0] https://www.theinformation.com/articles/secret-technique-beh...

What a clear and well-written article. I have only a basic understanding of LLM architecture and was able to follow along and gain intuition the whole time!

Anecdatum but I experienced looped cognition on a peculiar combination of substances.

I was able to treat thoughts as solid objects and manipulate them iteratively. (Ordinarily they're more like "glimpses" or "flashes" that fade rapidly. So I guess it would be like the mental equivalent of tracers.)

I was able to stack thoughts on top of each other, like planks. (I can do something similar or the narrowly but the planks are not nearly as wide!)

I didn't do any tests unfortunately but subjectively my cognition was greatly enhanced. (Spent a few years catching up with the insights I had that evening.)

Might be unrelated, but the part about "looped transformers" made me wonder if there's a similar "stepwise" increment going on here.

Edit: Okay, 6.8-18% is slightly less dramatic than what I was referring to.

I only used Astra while coding a bit so I can't comment on anything else but I have been really disappointed by it.

It seems to overengineer really bad and it is also very slow due to it "thinking" too much I feel like.

One example is that I asked it to implement a new functionality inside an existing App of mine and if I had written it myself it would have been like a ~50 line diff. Astra took like 10 minutes to write ~400 lines, most of them useless and also in pretty bad style, barely readable code.

Maybe I am bad with prompting but I didn't have these issues before, not even with 5.6 Sol on max reasoning.

This article is not up-to-date. There have been various benchmarks (some of which published and acknowledged by OpenAI, see the charts in this thread: https://xcancel.com/tomekkorbak/status/2095596839886274689) showing GPT-6 Astra is much less monitorable. The most recent third party benchmark I saw is showing a huge jump in capability for multi-hop reasoning without chain of thought: https://www.lesswrong.com/posts/FsCkkoGsNmPzFKRhg/gpt-6-astr...

I don't think this is explained by the model simply being more capable and therefore achieving more per token: the usage of recurrent depth (Neuralese) is exactly predicting less CoT monitorability even at equal capability.

On looped transformers:

previously, conversation might have 50k tokens spent on reasoning. the next turn takes all the previous tokens as well (if you wanna preserve prompt caching) which is not ideal. this new method skips that so you get more free context until compaction kicks in.

is this true? if so its a huge deal. why is it not spoken about? its one of the main reasons i don't use High or Max

Recommended reading from an actual researcher who thoroughly understands AI research papers and has an in depth analysis of models architectures and their mechanics and no nonsense benchmarks.

Probably off-topic. Astra has been kind of weird. Like, I can't trust it, weird. It has an interesting tone, especially in Codex, that is off-putting. It's over zealous at times (which is why I stopped using Claude) and gets too creative when doing agentic system level stuff. Accessing files and doing things it shouldn't do. If OpenAI was chasing Claude's approach, then they are going in the wrong direction. OpenAI has always been the "business and boring approach", which was its selling point and why I have stuck with it. Claude was always the radical one (powerful, but radical).

Also, Astra overlooked, in my opinion, a serious flaw in its approach for something I was working on recently, which really surprised me.

Reading between the lines, there were some breakthroughs with Astra, which I'm sure is why OpenAI released it so quickly after Sol, but probably not in the ways the traditional OpenAI customer wanted.

So the TL;DR here is that Astra's trick is that its a turbo-charged weaker model vs a larger heavier one-pass model -- and the turbo is instead of reasoning by 'talking out loud' and generating intermediary steps, the reasoning is able to be stored (probably as KV) and re-run as purely without the generation of the text. Making it more effecient to run successively and I assume more intelligent as the act of turning the KV cache into lingusitics loses some dimensionality (especitally spacially)

double TLDR: This is a Turbo V4 instead of a huge V8 of a model.

[flagged]