Artificial Analysis Intelligence Index v4.2

They realized Astra having the same score as Sol was silly so they rushed to update the index so it fits what people expect.

The old index was clearly bad (Astra is way better than Sol) but it's also unscientific to tweak it like this.

Imo the omniscience index they have has the highest correlation to actual usefulness of the models.

https://artificialanalysis.ai/evaluations/omniscience

> measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer.

This is so useful because it makes you actually trust a models output. A high score on benchmarks is not as useful because a model overtrained to always answer will give confidently wrong responses. But this index measures how often it is correct while penalizing wrong responses so that a high score means you can trust this model more and when it doesn't know it is more likely to tell you that it really doesn't know rather than making shit up.

Fable also performs a lot better than opus 5 here which correlates very strongly with perceived strength despite the models performing similarly on e.g. DeepSWE

Astra is a big jump from sol and performs the same or slightly better than fable here.

This is really a great achievement: "Astra dominates the output token frontier"

Many labs used increased thinking to boost benchmark scores and performance. Most of the Chinese models were doing that for a while. Google and Anthropic as well.

Not OpenAI. 5.6 already was much more token efficient than other models and Astra beats Sol in token efficiency by a wide margin.

Edit: Just to make the point: Astra (max) has the 2nd highest score and the third lowest output tokens (among the models shown by AA).

how did you check the prev version to compare?

This update really gives OpenAI a boost. Not saying there's anything inaccurate or untoward about that, but the timing is unfortunate. It would have looked better had it been done prior to the Fable 5.1 and GPT 6 releases. I guess AA would say that there's no perfect time to do these updates, given the rapid fire pace of releases!

Why have they not included ARC-Agi-3 on their index?

Clearly that would move things around.

What version the intelligence vs cost graph is using? they didn't ran v4.2 to all models.

How to view the previous version to compare?