logoalt Hacker News

tedsanders • today at 6:24 PM • 5 replies • view on HN

I work at OpenAI and I made the plots. Gemini 4 Carbon was omitted when we couldn’t find public data, usually cost. No intent to hide anything.

If we missed anything, feel free to point me to it and we can add it.


Replies

tasuki • today at 9:19 PM

> I work at OpenAI and I made the plots.

Have you not considered log scale for the "Agents' Last Exam v1.1" plot? Like this the left is so crowded I can't see a thing...

➕ show 1 reply
declanjackson • today at 7:32 PM

All TB-Science scores and Cost per Task (across all reasoning levels for Carbon) are on AA: https://artificialanalysis.ai/evaluations/terminal-bench-sci...

Would be great to see this on the plot as well

➕ show 1 reply
underdeserver • today at 6:36 PM

So all the plots are based off public data? You don't run the models internally yourselves?

(Again speaking only for myself)

pu_pe • today at 8:30 PM

Even OpenAI doesn't have access to Google's models

smurf9852 • today at 9:23 PM

Sorry Ted but your plots suck. Just look at that first one, Agents' Last Exam v1.1. All the data points are overlapping on the left side. I can't even see what is what