I disagree, I think distillation hides the true technical advantage since you can always get within X% of Claude by ex: training on newly created code on Github, but the underlying intelligence is harder to replicate.
based on the behavior of the chinese labs it seems like training on chain-of-thought is extremely important for distillation, and you can't get that by training on llm generated text.
based on the behavior of the chinese labs it seems like training on chain-of-thought is extremely important for distillation, and you can't get that by training on llm generated text.