is the open weights lag widening? these open models don't seem to be improving as quickly as ant/oai
I can't imagine them keeping up long term given the information asymmetry plus compute restrictions on top of it.
I feel open-weight was able to keep up the past year using efficiency improvements (hybrid attention, improved MTP drafters, improved optimizers) to offset having less compute. But OpenAI/Anthropic started cutting token costs just a few months after these, which to me suggests they copied these improvements. Indeed it would be hard to work in LLM training and not know about DSpark, Muon etc.
So what do you do if closed source uses your learnings, gives no knowledge back, and has 10x more compute and funding on top of it? The situation feels quite dim right now, esp. with the Nvidia thing.
This is Qwen4-Max, which is not open weights.
Their open weights models are the the smaller ones, which aren't released yet.
I think so but it may be a very short term effect, I think their pace has roughly matched the pre-speed up pace of the frontier labs (the 1.7-2x speed up noted on by METR), and I’d guess we see a similar speed up from the Chinese labs once their internal models hit an equivalent level enabling the same speed up.
Following that logic the gap widens a little bit here but then will steady again. Before this there has been a general catch-up trend too which probably also persists leaving us in a slow closing of the gap as the net effect?
Yeah, it turns out that the distillation theory has legs.
Might just be the commercial offerings scaling up parameters so much? Like for like the open models don’t seem to be lagging.
ant/oai have stalled in the past 6 mos,
No idea, but this is a rapidly changing field. Opus 4.5 was released a little less than a year ago and it feels shockingly limited compared to even current open flash models. (I literally just tested it the other day.)
So, with new models coming out on a weekly basis, I think trying to figure out a future trajectory is just going to be a wild guess. For all we know, Anthropic and OpenAI models may also be plateauing in the near future; or there may be a breakthrough and they may improve beyond our wildest expectations. I honestly don't know.
That said, I suspect available compute and model sizes that major Western vs. Chinese labs can handle will still keep the gap open at least in the short term.
On the other hand, I am also not sure how much the gap will matter past a certain point. I am specifically wondering at what point people will become interested in smaller models fine-tuned for specific application areas (e.g. cheminformatics) rather than massive generalist models.