alt Hacker Newsassume you are a "second class lab" and you are in fact making progress by distilling the results of the frontier labs' efforts.
what is the end game for this strategy?
if the frontier labs shut down, or stop releasing to the public, and there's noting left to distill, how will you progress?
Replies
Distillation from a teacher model solves the self-start problem, that is, building a model to the point where it reason coherently. Without distillation, solving self-start is incredibly difficult since it requires millions of high quality training samples. Creating that kind of dataset takes an enormous amount of effort.
Once a model becomes competent enough to perform complex reasoning, a teacher model is no longer necessary. The model can now reason about its own behavior and build a better version of itself through recursive self-improvement (RSI).
Kimi K3 is capable of RSI.
In public with budgets that don't risk destroying the American economy presumably. Yes it may be slower.
There doesn't need to be progress at this point. Some models even from 1 or more years ago are useful for some purposes
This line of thinking makes no sense because it assumes that labs that distill from frontier models are doing nothing else. It's the classic "the Chinese can only copy" mentality, and it's going to end poorly for American companies.
I'm pretty sure that all labs are distilling each others' LLMs, maybe apart from Anthropic and OpenAI. It would be stupid not to do it, because it's cheap and effective. But that's not the only thing they're doing. If you think K3 and GLM-5.2 got this good only from distilling frontier models, you're not paying attention to Chinese labs' publications.