alt Hacker NewsThe exclusion of harness's feels really weird given that companies are recognizing the value of what harness's can do. By excluding them the benchmark is becoming less relevant.
Replies
It’s because of inductive bias. Harnesses will massively skew results towards working solutions. You might think that’s a good thing but what it might mean that sometimes it becomes enough to run brute force search or a simple parameter search over the harness. Creating the harness is the actual work, because you’re selecting what are the levers to pull. There were some attempts of LLMs generating harnesses on the fly in ARC 2, but they were all mostly based on one handcrafted DSL that was copied over and over again. As it stands harnesses are not allowed because they’re simply not a meaningful measure. What you’d like is to measure how the model performs if it saw this benchmark for the very first time… but then again everyone knows the game is rigged, millions are at stake, and the AI companies fine tune and cheat on the benchmarks any way they can.
You could argue that if you allowed a harness, and that harness was specific for ARC, then you don’t have AGI, you have something that is definitely not general.
Harness can totally change capability of the model
iirc, a harness isn't allowed, but if the LLM wants to write its own tools to solve things that is allowed.