logoalt Hacker News

declanjackson • today at 7:32 PM • 1 reply • view on HN

All TB-Science scores and Cost per Task (across all reasoning levels for Carbon) are on AA: https://artificialanalysis.ai/evaluations/terminal-bench-sci...

Would be great to see this on the plot as well


Replies

tedsanders • today at 8:21 PM

One nuance to be aware of is that the AA version uses the mini-swe-agent harness, which can understate performance of models trained to use particular harnesses. Implementations also can differ in terms of timeouts on tasks and memory/etc. resources given to the agent, which affect scores on longer running, resource-intensive tasks. I believe these differences are why we chose not to use AA scores on this comparison, but I’ll double check when I get back to my computer.