This was also my thought, and I think holds true of the ones with invisible requirements that aren't stated up front and are only captured in tests. Oh, you need to rework your solution to handle requirements nobody mentioned before? Well, me too.

Yeah, it's testing a different thing than what the benchmark claims to test, but it's also accidentally testing something more real-world applicable than a clean benchmark would be, so hey.

(EDIT: That is, if the agent is allowed to see the failed tests and iterate. If not, then yeah, that's just a problem. And either way, the ones with tests that just encode a particular solution's implementation details, thereby demanding that your solution have some rando internal details, are junkier. That's not a situation you'd run into in reality.)