logoalt Hacker News

adrian_b • today at 7:52 PM • 3 replies • view on HN

For now I have not seen any credible evidence for "But chinese models use distillation of Frontiermodels...".

Of course, I would not be surprised if they also do this, mainly because after OpenAI and Anthropic have succeeded to gather most training information that was easily accessible in the past and now it has become much more difficult to do the same. For some information that was hoarded by OpenAI and Anthropic, public access may have become impossible otherwise than through their LLMs.

Nonetheless, distillation cannot be the main source of improvement for the Chinese models, among other reasons because even assuming that they may have access to a great number of fake accounts, there still is not enough time for a lot of distillation, taking into consideration the timeline of the releases, and it is not believable that the attempts to do distillation would not be detected quickly by OpenAI and Anthropic, leading to the banning of the fake accounts. While OpenAI and Anthropic provide negligible information about their models, the Chinese AI companies are much more transparent about their methods and we can see that there is a lot of diversity among their models and a lot of innovations with each new version.


Replies

rbehrends • today at 8:11 PM

Yeah, there is a limit to how much you can get from capturing reasoning traces. (Traditional distillation of weights is not possible via the API.)

Patrick Toulme noted a while ago that the biggest advantage of that in his opinion came from bootstrapping the post-training process: https://xxcancel.com/PatrickToulme/status/206921157543762774...

At Thinking Machines, they explicitly said that they used open-weights models for bootstrapping post-training, including Kimi K2.5: https://thinkingmachines.ai/news/introducing-inkling/

Speaking of Thinking Machines, that's another data point. Their Inkling Small 2 model is just a bit behind DeepSeek V4.1 Flash 1017 in benchmark, and in my tests, that seems to be accurate, but mostly because they used less training data for pre-training, so they have less of a knowledge base, while similar reasoning capabilities. (One of the most interesting parts of Inkling Small 2 is that their training must have been very cost-efficient, apparently using less, but higher quality training data.)

And we're now seeing more and more Western near-frontier models from labs other than Anthropic and OpenAI to question whether distilling reasoning traces is really much of a difference maker.

Personally, my theory is that the gap is primarily due to Chinese labs having a harder time getting the necessary compute (especially top of the line GPUs) for training. These are, as far as I can tell, orders of magnitude differences in availability, so it wouldn't be surprising that a large compute gap is reflected in capabilities (and also indirectly visible through model sizes).

origini • today at 8:00 PM

I don't think there is perfect proof, but its taken as truth.

Indication for this are for example that Chinese models think they are another model: "chinese model think its claude".

My assumption is, that Chinese model found things Frontierlabs already had but didn't share. There was this paper were i had the feeling that this was a real and significant Research Result from Deepseek "DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression" but otherwise its just unclear.

The main issue is, that as long as it seems plausable that they do distillation, its hard to determine how big the gap is.

Not saying they can't catch up but they are also potentially limited by resources (money & nvidia GPUs)

➕ show 1 reply
nzd • today at 8:04 PM

Heck, they destroyed some rare physical books after scanning them. Something to show their mentality.