I had a look at the changes and the biggest impact seems to be from adding Terminal-Bench-Science 0.1 and having it contribute 15% to the overall score.

Example problems are:

- https://hub.harborframework.com/tasks/terminal-bench-science...

- https://hub.harborframework.com/tasks/terminal-bench-science...

I have no doubt that these are very relevant task for people in those respective fields, but I suspect they have next to no relevance to some 99.9% of model users. Hence why especially smaller labs with smaller models didn't really train for such use cases and you can see it.

Personally, it's not something that matters at all to me, as I don't work in cheminformatics or bioinformatics.

This is also arguably more of a test of a model's knowledge base than reasoning ability.

But most importantly, there is also the question to which extent the aggregate score then remains a meaningful result when you mix up too many too disparate benchmarks.

The coding agents subindex, in contrast, looks at least superficially plausible [1].

[1] https://artificialanalysis.ai/agents/coding-agents#coding-ag...