alt Hacker NewsMan, a lot of this discussion sounds like people cheering for the last kid crossing the finish line.
Surely we want competition and Europe involved in that, but at this point I have grown used to either American labs smashing the frontier remarkably fast, or Chinese labs getting way, way closer than you would expect them to.
Mistral’s progress, regrettably, feels much slower. This model doesn’t knock anybody’s socks off. The model is (and I hate to be this harsh) mediocre, and this mediocrity has also arrived months late.
This is a pretty grim prognosis for European AI.
Replies
People were extremely dismissive of chinese models until recently. They went from 1 year behind frontier to 6 months behind frontier to 3 months behind frontier extremely fast.
Neat. Wait 3 months for the landscape to change entirely.
LLM development is jumpy. It’s hard to extrapolate very far ahead.
First few models will always be slow improving and worse. The way to improvement is working your way through a gajillion evals [1], finding bugs, gaps, and curating training data (this part involves human design as well as raw inference compute) to fix it. This is very time intensive and can't easily be "done once and then everyone has lesser work to do" since every model is different. Well, one way to accelerate it is to simply have more compute, which mostly openai and anthropic have[2].
This is mistrals first 1T-scale model and I expect the 4th or 5th generation to be close to the best for many purposes.
[1] These evals differ from the public ones like terminal-bench, are sometimes model-specific, need real, diverse usage to actually create, and are held secretly since quality of eval is the first driver behind the next step improvement of a model.
[2] It is not close. This model was trained on less than 4k GPUs, whereas astra used north of 100k GPUs.
Those are comments from Europe. The US is waking up now and I expect them to be much harsher.
I really want them to win as that's our last horse in the AI race, but ~200 research-oriented devs out of 1800 employees? I believe they agree it's pretty doomed and have pivoted.