alt Hacker NewsDishwasherBench – Can Frontier Models Stack a Dishwasher?
Comments
Can anyone confirm whether it simulates the spinning arm getting stuck on an item that juts downward through the top tray - thereby stopping the whole load from being washed properly.
Thats the most common problem in my dishwasher. It kindof looks like it would simulate that but they dont spell it out.
Come to think of it, if the players dont have an option to "spin the spinner by hand to make sure its not catching on anything" they're going to be in for a rough ride.
Inspired by Joma Tech's new YouTube video
Stacked loads are run against a dishwasher simulation calibrated to my own Bosch dishwasher
Playable by both humans and AI
Currently a WIP, simulations need refining before running a full benchmark, but keen to hear some feedback!
I love these novel ways to benchmark LLMs. I created one myself, I figured creating a programming language designed to confuse LLMs would be a good test, and it does seem to be: https://bench.killswitch-lang.org/.
I wonder if anyone has crafted a LLM benchmarks list website/github.
I have to admit that for many years I assumed a dishwasher physically filled the entire chamber with hot water to wash dishes. Then, one day a person opened up a dishwasher while it was running, took out a plate, clicked start, and closed the washer. My mental model literally changed in an instant.
Perhaps a better question would be "can frontier models write a program to effectively stack a dishwasher?"
Years ago after my wife filled the dishwasher I commented that it looked like a 12-year-old filled it. I said it as a joke but it really hurt her and I still feel bad about it. That said, it no longer looks like that when she fills it...
I love this! Cant wait to try this.
This is so cool!
Imagine creating real simulated env for every physical task and using RL to train robots. That will be a game changer.
Both models’ loads came out 84-100% clean, so is the score mostly just how much they fit?
The real story of how Navier Stokes was solved involved a dishwasher:
For added difficulty, have it 'stack' a 3D rendered volumetric plastic model of a pelican riding a bicycle, like if it were putting a child's toy in the dishwasher.
[dead]
[flagged]
The obvious fork is "can frontier models stack a dishwasher to my wife's exacting standards".
I can't. So that would surely be proof of AGI.