This was also my thought, and I think holds true of the ones with invisible requirements that aren't stated up front and are only captured in tests. Oh, you need to rework your solution to handle requirements nobody mentioned before? Well, me too.

Yeah, it's testing a different thing than what the benchmark claims to test, but it's also accidentally testing something more real-world applicable than a clean benchmark would be, so hey.

(EDIT: That is, if the agent is allowed to see the failed tests and iterate. If not, then yeah, that's just a problem. And either way, the ones with tests that just encode a particular solution's implementation details, thereby demanding that your solution have some rando internal details, are junkier. That's not a situation you'd run into in reality.)

Agreed - "underspecified prompts" being listed as a failure of the tooling is not a strong case. Even interns can understand ambiguous asks with a bit of help, and understand when they need to stop and ask instead of just carrying on. They are often working fairly independently on ambiguous tasks before the end of an internship, too.

So is the argument that frontier models are not just junior engineers, but first-month interns with no capability of progressing beyond that level?

In a real job, you would be allowed to see the test case that failed and tweak your code (or more likely the poorly written test).

If you let a modern LLM do even the first, they’d crush this specific benchmark.

What is interesting is understanding how LLMs are able to beat 70+% on this benchmark or getting some of the poorly framed questions right? Are they implicitly learning the test writers style? Are the solutions leaking into their training set?

Perhaps reassuring is that even Fable stalls out at ~72% (on the hidden set which OpenAI did not run this analysis on), so perhaps training on the bench is not happening in anything but the most indirect ways.

I care a lot because small open models can never learn idiosyncrasies like this, so I really want good ways to judge models fairly.

EDIT: Humm OpenAI is muddying the water a bit. Only 20%ish of problems are broken in ways that are unfair to the agent, 4-10% are broken in favorable ways, so the benchmark ceiling is probably closer to 80-85%

The more subtle point is that there's a gap between the task and its verification. e.g. if you have an open-ended / under-specified prompt, the verification needs to be able to handle all potential solutions.

So you can have a very narrow task prompt that's easy to verify (but likely too simple of a challenge). Or a more realistic task prompt that's much harder to verify. And likely harder to both build the robust verifier and run it cheaply.