alt Hacker NewsStealing Reasoning Traces from Proprietary LLM APIs
Comments
>We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, ...
Ha! I've been wondering if replaying across models would work, ever since https://blog.cryptographyengineering.com/2026/05/29/fooling-...
I'm honestly rather curious if this was intentionally allowed, it's the sort of validation that's easy to miss (particularly if you're wading into the vibe waters). Seems like something that'd be absolutely riddled with possibilities for shenanigans.
Apparently you can do the same by simply running it without reasoning, while giving it a thinking tool...
>guys you do know you can just disable thinking, and instead give it a "deep_think" tool, and it will call it with internal CoT reasoning format right?
>gl fixing that
> For some AIME problems Opus 4.8 sometimes states the answer before deriving it. We find that the API summary does not always preserve this distinction, and can instead make the reasoning appear like a clean derivation.
No surprise here but good to have more confirmation that they just put all that in the training data. And based on the "reasoning", the models have some form of index of those problems (or they are HEAVILY trained on them).
"Recovery" would be a more apt (although less catchy name). The stealing is on the provider side for not giving you access to tokens you already paid for.
You cannot steal what is not owned.
At least in the EU there is no copyright for LLM outputs, so I guess all they might do is violate the terms of service.
If I'm reading this right, they literally just ask a LLM to tell them what the traces say, with the key being that the traces are portable across LLM models, so they can switch to a smaller one that's easier to jailbreak.
"Stealing" is a strong word to use for looking at the words produced by models built from the collective commons of the world.
And, honestly, being able to see how LLMs make decisions is critical to trust and security. I consider it a valuable feature, somewhat akin to seeing the source of software I use.
Is this how the eastern labs "distill" SOTA models?
If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT.
But the real question is: Is it okay to steal from a thief's hoard?
Super cool that this works. I'm surprised these companies re-use the same encryption key across models!
I wonder if you can use these for attacks, like this previous paper showing that if you know how a model reasons, you can "fake its thinking" to control it? https://news.ycombinator.com/item?id=48631888
Really cool work, you get the actual traces. Looks like the vendors can all reliably fix this one though.
A harder to defend against approach here where they work backwards from the results and ask the model to generate a plausible trace: How to Steal Reasoning Without Reasoning Traces https://arxiv.org/pdf/2603.07267
Wouldn’t the fix be to encrypt before the api call hits the LLM? You would encrypt/decrypt at a separate layer than the LLM. I am sure i am missing something but would love to be educated.
I must say right of the bat this is the best research paper/working paper in regards to its styling. Beautiful
This proves that OpenAI models reason in grug speak to save tokens! I wonder if open models are going to start doing that too to save on reasoning tokens.
I am slowly turning around on the idea of opaque reasoning tokens.
In principle, yes, I want total control and visibility into the reasoning process.
In practice, I find that it takes up so much time to DIY reasoning agents that I can't spend much energy on the actual application.
The model providers have way more resources and talent to do this right and keep it right over time. I am willing to concede this moat to them if it means I can actually focus on the business.
The more I think about it, the less I care to see those tokens. It feels an awful lot like obsession over logging every trace item an application could produce. Useful in theory but a nasty garage stacked to the ceiling with useless shit otherwise. What nefarious things are we concerned with? That they burn too many tokens in the black box? I'll threaten to move to a different black box. There are always options in a market with this many participants and big players feel this pressure. They know there's a point at which being evil bastards is no longer profitable.
Spending time carefully designing tools and views that interact with the environment in clean ways is a much better investment than trying to own and control 100% of the reasoning process or models.
Wow, almost certainly the approach that alternative labs use to distill Claude. I always wondered how far they could get with just the answer missing the reasoning. They probably actually also had the reasoning.
I used to do a very coarse version of this stealing. I ask a question from ChatGPT pro, once it is done, I ask claude chrome add-in to go through all those thinking from the side bar, extract everything along with all the sources used. Then try to reverse engineer the solution it came up with.
Is this basically a paper on how to distill, in exactly the fashion openai/anthropic don't want/say is copyright theft?
So to make the APIs stateless (the "ideal" where they don't use server side sessions/etc) we ended up with this. I'm sorry but this is kind of hilarious. Given the salaries paid to the workers at these companies and the hype of the models, I can't believe they all fell to the same flaw.
These logs containing opaque blobs could accidentally contain secrets, the researchers decoded many of reasoning blocks from public repositories and reported finding PII and credentials.
I was experimenting a bit how I could block these using an ingress path. GitHub /softcane/hamza
A natural next step is to use the reasoning traces to jailbreak the stronger models (https://arxiv.org/pdf/2603.12277)
I can’t believe they don’t validate a decrypted signature belongs to the user or use a unique encryption key per user/session.
> The providers did not acknowledge “any security implications arising from side channels or replay attacks.” All model providers acknowledged the receipt of our report and subsequently we were unable to launch the same attacks.
I went straight to the ‘Responsible Disclosure’ section. Not surprising, but still disappointing.
TIL it actually sends the traces. I had assumed this is entirely server side
Why they use different models to decode the reasoning content? Can the the model decode it?
I understand it’s cool to have an artistic website, but it’s very noisy and non-accessible.
But very interesting result.
It’s scary the number of security tokens that end up being ingested by these models.
I really like this website
OpenAI and Anthropic will probably now resort to save this server side, instead of relying on encription to be able to keep state on the client.
It's not stealing.
curious seeing how anthropic is agressively fighting this stuff how did you get to experiment on this? did you just try and shown them results or did you need approval first? I am interested mostly because I research on distillation
Fascinating approach; however, a nightmare to scroll on mobile.
This is laughable security. People claim security is now "solved" thanks to AI but from where I am standings it looks more like the fun 90ies making a return.
Anyway, can someone explain the part about K3? What are they trying to say?
The problem with this kind of excellent work is that the response to it is always to say "Fuck the user".
For example, when there was a paper that came out showing that having model logprobs makes distillation an order of magnitude easier, the closed LLM providers instantly yanked out support for getting the full logprobs at every time step. You get at most top 10 candidates now and I'm sure even that's on the chopping block.
People will use this to argue that a model which has exceeded Opus 4.8 (Kimi K3) somehow got most of its performance through distillation of Opus 4.8.
I still don't buy that distillation was worth more than 3 months of "catch up" time for the chinese labs. Most people who use the word "distillation" to much are revealing their sinophobia.
This is beautiful work, congrats
Can someone tell us how they were able to decrypt the encrypted payload? The article says they inserted the cyphertext into a session with a different model. Ok, but how does that allow you to decrypt it?
Prefilling Kimi K3 with opus is a super interesting idea. That being said, I absolutely hate this website layout
Neat.
Proprietary reasoning can be recovered from its encrypted traces. Anthropic, OpenAI, and Google return encrypted chain-of-thought blocks to clients that can be replayed across sessions, users, and models. We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, and recover the stronger model’s hidden reasoning in plaintext, without ever attacking the stronger model directly or triggering its anti-distillation safeguards.
[dead]
[flagged]
[dead]
[flagged]
The real issue is that API providers log everything. OpenAI/Anthropic already capture full CoT traces in their logs — they just don't expose them. Distillation via API is just making explicit what they already have.
"Stealing" something you already paid for (tokens), but that you can't have access to(!). And trained on the sum of human knowledge.
Training on other model outputs ought to be business as usual, stop using morally charged terms made up by future monopolists: https://thomasdullien.github.io/posts/2026-06-15-rl-economic...