logoalt Hacker News

tedsanderstoday at 6:44 AM2 repliesview on HN

I work at OpenAI and can confirm that's correct: reasoning tokens are discarded after each new user turn (though not after each message or tool call).

Our docs show a diagram here:

https://developers.openai.com/api/docs/guides/reasoning

> Input and output tokens from each step are carried over, while reasoning tokens are discarded.

Keeping reasoning tokens around is better for caching and for remembering past insights, so you might reasonably wonder why we designed it this way. The main benefit of dropping reasoning tokens is that you can fit a lot more work inside the model's context window before you're forced into a slow and lossy compaction step. This was a larger consideration with our earlier reasoning models that had shorter context windows (~200k), longer thinking times (up to ~100k per message), and poor compaction. However, now that we've shipped longer context windows, we've trained our models think much more efficiently, and we've made compaction way better than it used to be, the balance of factors is changing. Tune in Thursday!


Replies

CjHubertoday at 8:16 AM

> fit a lot more work inside the model's context window before you're forced into a slow and lossy compaction step

This is something I never understood. Why the reasoning is not included until the context is full, then the reasoning stripped optionally to allow the conversation to continue. and only then when its truly full offer a compaction. Was it to optimize caching? Well I guess it doesn't matter now that you hinted that this choice was made because of prior limitations and may change very soon

show 1 reply
throwuxiytayqtoday at 6:48 AM

dude what the hell