Would be good to see the full details of the tasks / methodology used. https://eebench.org/methodology.html makes this unclear.

> A real capacitor makes the task more interesting. A ceramic part may provide much less than its advertised capacitance once it has voltage across it. Parts have tolerances. Adding more capacitance costs more, takes up space and makes the rail slower to recharge when the power returns. A design that works with nominal values can fail with the parts that arrive.

It sounds like from a reasonable reading of the benchmark post that there's some things that are being tested that are assumed to be criteria that you expect the models to intuitively find those things to be important (i.e. the stuff about working on parts that have tolerances etc.). If that's so, then this really feels like mostly an exploration of whether an LLM has a good understanding of unstated constraints and has an appropriate in distribution set of priors that would be able to form models where it's reasonable to design on those lines.

It's hard to tell whether this is a problem though as the methodology is imprecise.

If you're spending time on evals against your own product, I'd be super curious to see how far you can get to by using a top tier model to produce generalized instructions for lower tier models. E.g. in a loop: "This eval missed X. what's the simplest single instruction that would have helped this session consider that as necessary that can benefit all future runs. Stick that in AGENTS.md and retest."