logoalt Hacker News

intenex • today at 5:54 PM • 23 replies • view on HN

Performance is insane for the cost if those benchmarks hold up. Looks like they did time this for right before Anthropic's IPO. Curious what Anthropic releases in response, and how these do on real world tasks.

Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?

Edit: Wow, interesting very negative vibe check here: https://x.com/mattshumer_/status/2122752192226406421?a=fVRt3...


Replies

mlinsey • today at 6:30 PM

If you have the quota or budget for it, I recommend vibe-coding your own personal benchmark. I have a suite of 8 projects that I made with models this year, and replay them - give the new model a checkout of the repo at that point, and the project spec. For the design task, I have it throw up a blind voting page for me to quickly judge, and for implementation I have a review agent do a general score and also hunt for problems/edge cases that I really found.

It's a lot more imprecise than the public benchmarks, but it's tailored to my own work and definitely isn't being benchmaxxed by the model developers. I was able to tell really quickly that Opus 5.5 delivered on the hype, and that some other releases (a couple of the more recent open releases, and pre-Carbon gemini models) were not as good for me as public benchmarks made them look.

(no results for either 6.2 yet, will take another ~6-8 hours, but I encourage people reading this with spare tokens to just make their own like I did instead of listen to me)

➕ show 6 replies
alastairr • today at 6:03 PM

I didn't find Sol 6.1 nearly as impressive in real SWE use compared to what the benchmarks implied. It wouldn't surprise me if the models have internalised some kind of 'benchmarking mode' which only kicks in when some attributes that are peculiar to how benchmark problems are presented.

➕ show 10 replies
Kostic • today at 9:54 PM

You should be aware that the individual you shared had a project called "Reflection 70B" that was, to put it mildly, quite disappointing and misleading.

revolvingthrow • today at 6:11 PM

> Looks like they did time this for right before Anthropic's IPO.

Would it be in OAI's interest to kick the legs from under Anthropic? Granted I don't think they can, but a shot across the bow can sometimes be more damaging that one expects (I think FTX collapsed due to one? Or maybe I'm misremembering).

I always assumed that if one of the big AI IPOs fail hard enough the neoclouds start to go bust, which causes a chain reaction that probably causes both ANT and OAI to collapse.

➕ show 5 replies
platinumrad • today at 7:05 PM

Shumer is a known scammer (Reflection 70B, among other things). Why does he have any credibility?

➕ show 4 replies
stbenjam • today at 6:56 PM

My vibe bench is no dependency HTML bowls of ramen https://bitbin.de/ramen-bench/

Opus 5.5 Max is my favorite, but some of the Sol/Astra 6.2 ones are pretty good.

Edit: some folks reported crashes. It was probably Gemini 4 Carbon @ High, I've put it behind a warning. If you see another one let me know (OS, browser, which bowl, etc)

➕ show 3 replies
xtracto • today at 6:43 PM

>Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?

Generate an SVG with complete and detailed graphical instructions on how to fold an origami pelican riding a bicycle

(or any other complex origami object). The fact that it has to teach [you] something makes it pretty interesting.

imjonse • today at 8:57 PM

that negative vibe check is from the Reflection guy who in 2024 wrapped Claude API and presented it as some new breakthrough open weight model of his own.

the_duke • today at 6:29 PM

> Looks like they did time this for right before Anthropic's IPO.

While both Anthropic and OpenAI have been playing games with model announcement timings, this time I don't think it's related.

These are incremental bumps, not big new models that have a huge impact.

giancarlostoro • today at 7:42 PM

Well, Fable also went down in cost too in their last update to Fable, and honestly, if they released a crazy new model on IPO day it wouldn't surprise me. Save the best for an impressive IPO.

karimf • today at 6:18 PM

I'm very curious about this as well.

Recursive self-improvement (RSI) has been a very hot topic recently, and it could be the biggest factor that decide Anthropic's IPO.

I wonder if the general public can infer whether RSI has started based on the recent performance gain of the frontier model.

➕ show 3 replies
bob1029 • today at 7:17 PM

gpt-6.1-terra is already beyond end game for my client's business. They don't require anything more powerful.

Cost is the most important factor now.

wolttam • today at 8:54 PM

Matt Shumer says things to get attention, and almost no other reason.

Anybody remember https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B

"Oh, the quality is bad because we uploaded the wrong weights."

i_have_an_idea • today at 7:04 PM

An Anthropic shareholder is very negative about an Anthropic-beating model release, right before the Anthropic IPO? Interesting, yes :)

6thbit • today at 6:44 PM

They're going after anthropic's market through cheaper intelligence per token.

I wouldn't be surprised if they released 6.2 luna with the most attractive positioning on the pareto frontier right as the IPO happens.

Thrombocius • today at 7:34 PM

Thoroughbench Fireship’s horse animation test is actually a pretty nice vibe check.

ijustlovemath • today at 6:00 PM

I've been having fun with having it build interactive Three.js demos of devices, with setup screen, a few different operational modes, mechanical movements etc

➕ show 1 reply
kylehotchkiss • today at 7:39 PM

Sincerely curious: why do people feel led to use the word "insane" to describe every new model release? I just don't understand what about incremental improvements needs to elict such a strong response.

We knew that improvements would continue to roll out increimentally, the costs would go down, we watched Moores law for decades before this?

➕ show 1 reply
ranyume • today at 6:26 PM

You're looking for a vibe check?

crimsoneer • today at 6:58 PM

I'm just running it on my Blood Bowl Bench, holding it's own against the reigning champ Argon so far but early days...

https://botbowlbench.com/

teknologist • today at 7:19 PM

Beware of the paid influencers who don't disclose their affiliations in this space. Theo is another one who now talks smack about anything that isn't Claude. They bend reality to fit a narrative.

aleqs • today at 7:12 PM

None of these benchmarks/marketing numbers from openai or anthropic have ever held up.

Now that we know anthropic nerfs models after release, and without any notice/openess, there is no reason to even think these benchmarks they release are even done with the same model/intelligence that actual customers would get.

behnamoh • today at 6:39 PM

> Feels like we're way past the pelican benchmark being useful at this point.

It was neither a benchmark nor a good one at that. Just a blogger's curiosity that grew out of control.