logoalt Hacker News

Astra 6.2 and Sol 6.2 flop at Simple Bench

52 points • by algoth1 • today at 1:16 PM • 35 comments • view on HN

Comments

ricardobeat • today at 3:13 PM

Neon / Carbon / Argon / Hydrogen are all in the same ballpark, why did they even release four different models?

Neon doesn't seem particularly strong at coding vs Argon, and Opus 5.5 still beats both in my experience. The neurons it lost that made it incompetent at English seem to have been fully dedicated to being a freak at engineering.

➕ show 1 reply
davidmurdoch • today at 3:14 PM

I've been finding 6.2 Sol doesn't act like an architect and doesn't doubt its instructions or existing code at all. It's back to "you're right to push back" responses when questioned about it's plans/reasoning in the slightest.

0xbadc0de5 • today at 3:02 PM

Having used both Sol-6.2 and Astra-6.2 heavily over the last few days (as well as regularly using Claude Opus and Fable) they are marked improvements over the previous releases and comparable to Opus-5.5 and Fable-5.2.

I've stopped trusting any of these influencer-benchmaxxers because they generally don't run any of the workflows of interest to me - it's completely synthetic. The only benchmark that matters is how well any model works for your current use-case.

➕ show 1 reply
martinjc • today at 2:59 PM

For my experience, i feel it follows instructions too well. Grey areas are completely cut out, and sure thats good sometimes. But for my usecase, i liked when the response felt smart and ready to ask questions.

algoth1 • today at 1:16 PM

Phillip has updated the leaderboard and surprisingly every new model release got substantially better, except for the OpenAi models. 6.2 Sol is 30th whereas 6.1 Sol is the 10th. Astra 6 is 9th, Astra 6.2 is 15th. By comparison Fable 5.2 became the 1st and Fable 5.1 is 7th. All the new google models, even hydrogen, are above the human baseline. By contrast, Astra 6 pro is the only openai model slightly above the human baseline at 8th place

➕ show 1 reply
TrackerFF • today at 3:13 PM

Makes me wonder if people are getting different models?

So far, my experience with 6.2 is that it is crazy fast compared the before, and much more to the point.

hoppp • today at 2:59 PM

All the new Gemini models perform better on this than Opus 5.5

Is that something that can be noticed during real life use?

I really like opus 5.5, It's perfect and I haven't tried the new gemini models.

➕ show 4 replies
Culonavirus • today at 2:57 PM

6.2 is for sure the worst received update OpenAI has ever released.

What happened lol.

I wanted to update our older used 5.x models in api to 6.2 the other day (due to lower cost), asked Fable about the difference between 6.1 and 6.2 if the cost is the same and it was like "yeah don't use 6.2, it has bad feedback online, has increased laziness etc., use 6.1". Yikes.

➕ show 3 replies
delichon • today at 3:20 PM

> SimpleBench includes over 200 questions covering spatio-temporal reasoning, social intelligence, and what we call linguistic adversarial robustness (or trick questions).

We could create a utopia if we could screen voters for linguistic adversarial robustness.

arthur-st • today at 3:08 PM

This is a strange benchmark. Gemini Flash 3.8 outperforms Opus 5 and Opus 4.8, and the baseline seems to be calibrated against responses of 9 people.

bieganski • today at 2:40 PM

I tried taking the test, but ultimately stopped at Question 4:

There are two sisters, Amy who always speaks mistruths and Sam who always lies.

:)

➕ show 3 replies
goodcake3963 • today at 3:00 PM

Google comeback is very interesting, after all they are the healthiest company in the field with in-house hardware

➕ show 1 reply
varispeed • today at 3:07 PM

I wonder if OP got routed to the nerfed versions. 6.2 works excellent for me. Getting better results than 6.1 and faster. On the other hand Fable and Opus are a disaster. The stupidity of those models reminds me of early Gemini.

xqcgrek2 • today at 3:06 PM

As the kids say, OpenAI is cooked.

Terrible funding rounds, uncompetitive models, enshitification, and insurmountably bad and incompetent PR.

floornext • today at 2:43 PM

[dead]