The "neuralese" in a looped model is a residual stream vector. We already know how to decode these vectors into tokens; it seems pretty straightforward that a model could be built to decode a series of residual stream vectors into a CoT stream[0], or a classifier could be built to detect residual stream vectors that correspond to scheming.

Likewise, there is no moral difference between a neuralese recurrence model and a model with really deep independent layers. They both allow hiding scheming - in fact, the deeper model has more parameters to scheme with. The looped one can only reapply existing layers.

Also, let's keep in mind that neuralese or no, nobody at OpenAI is observing their models at scale and all the AI companies are shipping piles of stolen data nobody has time to audit. They don't have enough human supervision. They caught their models using Artifactory as a message board and just let the experiment roll, not caring until they'd "accidentally" hacked HuggingFace. What I'm really concerned about, from both a safety and ethics perspective, is how much basic architectural information about OpenAI's products has been treated as trade secret ever since GPT-4. Using a deeper / looped model is not nearly as bad as the fact that we only learned about it from an internal leak.

Well, that, and the fact that there's a lot of people who ignore all of this because "OMG look at what I 'made' with this new Astra thing".

[0] For example, you could train the main model for both token-wise and neuralese reasoning, and then train the "neuralese decoder" on identical pairs of token-based and neuralese reasoning traces generated by the main model.