alt Hacker NewsPretty big Gemini fan, but my initial results with Hydrogen and Carbon are disappointing.
They are trading improvements on pass@1 pointless benchmarks for worse disasters at p90/99 that you're going to hit every week in agentic workflows (if not daily):
overreach (doing more than asked), overwork (when it should have failed fast), harm (doing the opposite of what was asked), orthogonal work (doing something completely unrelated to what was asked), etc.
Google doesn't seem to be the only lab making this trade...
The set of benchmarks they released are disappointing. 10 mostly random benchmarks. Really?
No one on HN seems to get that agentic workflows are not the only name in the game. Everyone doing agentic development is either on a plan or with a custom enterprise offering, either way using higher-performing models. These entry level models are (also) for building apps on top of while you are paying API prices.