logoalt Hacker News

furyofantares • today at 8:28 PM • 1 reply • view on HN

Would local models catch up if frontier models paused new training for a year?

I'm not sure for. It might just be you can get these nice local models because of distillation from a much more powerful model.


Replies

swiftcoder • today at 8:36 PM

Adversarial distillation hasn’t been a major factor in the past year. If the SOTA stops moving, all the other labs catch up to them in 6 months to a year, and distillation into smaller models continues to close the gap between small/large models during that time