alt Hacker NewsSeems like depending on your field these days, the hot thing to do is build your own private benchmarks.
In my own testing, no frontier model knows how to replicate an original 1990s Super Soaker prototype design, which for the most part, should be almost completely possible with Home Depot parts.
They just don't understand PVC parts, triggers, etc.
Replies
softwaredoug • yesterday at 9:20 PM
Or defensively expect models to be stupid.
Seems the smart thing to do is not assume an agent will do the right thing. But to create the scaffold / harness that enforces constraints to steer them towards a good result.
Then you can swap out the really smart model for maybe something cheaper.
thierrydamiba • yesterday at 9:25 PM
Or you’re getting steered into la la land because of your prompt
➕ show 1 reply
It's a "commonsense spatial reasoning/problem solving" kind of problem. LLMs fail at spatial reasoning forever.
What humans "easily" solve in seconds with raw spatial reasoning LLMs often find easier to solve by invoking A* or a constraint solver.
Might be that text data is particularly bad at teaching that to LLMs. Or that being good at spatial reasoning requires true recurrence, and autoregressive chain of thought is a poor substitute. Or it might be that human brain was optimized by evolution for solving spatial problems in open ended 3D environments for hundreds of millions of years, optimized for language for mere hundreds of thousands of years, and only optimized for writing computer code for a few decades at most.
The current frontier is halfway competent at benign closed 2D work, but still completely fumbles anything remotely close to open ended real world 3D work. It's getting better, but very slowly.