alt Hacker NewsI used to think that way about SVGBench, after all labs can just train on the test set, right? It turns out the task was highly generalizable. Try designing a logo and you quickly see the gap between models visually. Even though there is still a gap between "shiny demo SVG" and actual real-world use.
Same thing happened with MineBench, basically SVGBench+3D, until that got "saturated".
Remember spinning hexagon bench[1]? Or the AI World Clocks[2]? Yeah, that used to be hard for frontier models.
Creating games is the next iteration that still has some signal left. Assets + Game logic + UI + Sound, it let's you assess a model's "taste" very quickly.
What else is left, once all these benchmarks get saturated?
> What else is left, once all these benchmarks get saturated?
I was thinking this thing quite recently and wish to write a blog post about it but IMO, I believe that the next AI benchmarks would be profitable businessmaking. There are/were already some trials done (The famous WSJ/Anthropic one[0]) and some shop iirc in SF which is doing this and recent example of trying to start a lemonade stand with AI[1]
In my opinion, it can't really get more meta than that plus on the more humour side, it could help these companies make some dollars as they desperately need it.
I always wonder if these AI models can self-autonomously generate profit, why would these AI companies try to let you generate the main profit while you just pay them one time or a very small amount in tokens. Why should these AI giants allow a person to be a middle man in the first place in some sense and instead not just run these agents autonomously.
Maybe it could be because these AI agents might not be the best in such a meta task (money making/business) but as such, it does feel to me that it might be a good benchmark in the first place.
[0]: We Let AI Run a Vending Machine. It Lost All the Money. | WSJ: https://www.youtube.com/watch?v=SpPhm7S9vsQ
[1]: Everything AI Does When You Ask It to Start a Lemonade Stand: https://www.youtube.com/watch?v=6Ide5pRLR8Y