logoalt Hacker News

ksec • today at 9:09 PM • 2 replies • view on HN

As someone who dont follows closely. How do I know at what point were the other GPT model tested?

Let say they just nerfed GPT 6.1 a few weeks ago. Now they are using that as a reference score against 6.2? Or were those results tested when 6.1 relesed. What about 6.0?

And that is the same with Claude. How do we know they are not cherry picking the date and test on these release when they are constantly "enhancing" their model.

We also know some of these benchmarks are largely irrelevant once they were known ahead of time and become a game of benchmaxxxing. It just follows Goodhart's Law "When a measure becomes a target, it ceases to be a good measure."

So how should I know what has actually improved?


Replies

therealdrag0 • today at 9:11 PM

There are benchmarks that retest over time and look for nerfing. iirc they disconfirm the vast majority of suspicions about this.

➕ show 1 reply
jbellis • today at 9:22 PM

I've been testing sol, and opus, almost weekly as new models are released. I haven't seen any evidence of the nerfing conspiracy theories. https://slopcop.com/power-ranking

Edit: But it does look like you get about 20% less tokens for your subscription with OpenAI than you did until a couple months ago.

➕ show 1 reply