logoalt Hacker News

rbehrends • today at 8:11 PM • 0 replies • view on HN

Yeah, there is a limit to how much you can get from capturing reasoning traces. (Traditional distillation of weights is not possible via the API.)

Patrick Toulme noted a while ago that the biggest advantage of that in his opinion came from bootstrapping the post-training process: https://xxcancel.com/PatrickToulme/status/206921157543762774...

At Thinking Machines, they explicitly said that they used open-weights models for bootstrapping post-training, including Kimi K2.5: https://thinkingmachines.ai/news/introducing-inkling/

Speaking of Thinking Machines, that's another data point. Their Inkling Small 2 model is just a bit behind DeepSeek V4.1 Flash 1017 in benchmark, and in my tests, that seems to be accurate, but mostly because they used less training data for pre-training, so they have less of a knowledge base, while similar reasoning capabilities. (One of the most interesting parts of Inkling Small 2 is that their training must have been very cost-efficient, apparently using less, but higher quality training data.)

And we're now seeing more and more Western near-frontier models from labs other than Anthropic and OpenAI to question whether distilling reasoning traces is really much of a difference maker.

Personally, my theory is that the gap is primarily due to Chinese labs having a harder time getting the necessary compute (especially top of the line GPUs) for training. These are, as far as I can tell, orders of magnitude differences in availability, so it wouldn't be surprising that a large compute gap is reflected in capabilities (and also indirectly visible through model sizes).