I've been finding 6.2 Sol doesn't act like an architect and doesn't doubt its instructions or existing code at all. It's back to "you're right to push back" responses when questioned about it's plans/reasoning in the slightest.
Having used both Sol-6.2 and Astra-6.2 heavily over the last few days (as well as regularly using Claude Opus and Fable) they are marked improvements over the previous releases and comparable to Opus-5.5 and Fable-5.2.
I've stopped trusting any of these influencer-benchmaxxers because they generally don't run any of the workflows of interest to me - it's completely synthetic. The only benchmark that matters is how well any model works for your current use-case.
For my experience, i feel it follows instructions too well. Grey areas are completely cut out, and sure thats good sometimes. But for my usecase, i liked when the response felt smart and ready to ask questions.
Phillip has updated the leaderboard and surprisingly every new model release got substantially better, except for the OpenAi models. 6.2 Sol is 30th whereas 6.1 Sol is the 10th. Astra 6 is 9th, Astra 6.2 is 15th. By comparison Fable 5.2 became the 1st and Fable 5.1 is 7th. All the new google models, even hydrogen, are above the human baseline. By contrast, Astra 6 pro is the only openai model slightly above the human baseline at 8th place
Makes me wonder if people are getting different models?
So far, my experience with 6.2 is that it is crazy fast compared the before, and much more to the point.
All the new Gemini models perform better on this than Opus 5.5
Is that something that can be noticed during real life use?
I really like opus 5.5, It's perfect and I haven't tried the new gemini models.
6.2 is for sure the worst received update OpenAI has ever released.
What happened lol.
I wanted to update our older used 5.x models in api to 6.2 the other day (due to lower cost), asked Fable about the difference between 6.1 and 6.2 if the cost is the same and it was like "yeah don't use 6.2, it has bad feedback online, has increased laziness etc., use 6.1". Yikes.
> SimpleBench includes over 200 questions covering spatio-temporal reasoning, social intelligence, and what we call linguistic adversarial robustness (or trick questions).
We could create a utopia if we could screen voters for linguistic adversarial robustness.
This is a strange benchmark. Gemini Flash 3.8 outperforms Opus 5 and Opus 4.8, and the baseline seems to be calibrated against responses of 9 people.
I tried taking the test, but ultimately stopped at Question 4:
There are two sisters, Amy who always speaks mistruths and Sam who always lies.
:)
Google comeback is very interesting, after all they are the healthiest company in the field with in-house hardware
I wonder if OP got routed to the nerfed versions. 6.2 works excellent for me. Getting better results than 6.1 and faster. On the other hand Fable and Opus are a disaster. The stupidity of those models reminds me of early Gemini.
As the kids say, OpenAI is cooked.
Terrible funding rounds, uncompetitive models, enshitification, and insurmountably bad and incompetent PR.
[dead]
Neon / Carbon / Argon / Hydrogen are all in the same ballpark, why did they even release four different models?
Neon doesn't seem particularly strong at coding vs Argon, and Opus 5.5 still beats both in my experience. The neurons it lost that made it incompetent at English seem to have been fully dedicated to being a freak at engineering.