alt Hacker NewsI found that Google does not benchmaxx as much as the other providers. Of you look at real-case evaluation like lm-arena, even the 3.8 flash is often near the top despite its benchmark index being worse.
I found that Google does not benchmaxx as much as the other providers. Of you look at real-case evaluation like lm-arena, even the 3.8 flash is often near the top despite its benchmark index being worse.
I noticed this recently with the user submitted benchmarks on Kaggle. For these obscure tests that the models haven't seen, Gemini 3.8 is often on par or beating other frontier models. Gemini is a bit shite at the game checkers though, for some weird reason it performs poorly on those benchmarks.