alt Hacker NewsThis is the first I have herd of this benchmark. Can someone explain how it in any way indicates how close we are to "AGI"?
Replay of Sol attempting the game: https://arcprize.org/replay/83543d22-8e1e-439a-8809-129ff1d9...
It seems a weird and arbitrary challenge for a language model to be expected to perform. It also seems like there are some harness/visual issues even in the first few steps, where it states that it hasn't moved when it clearly has.
the problems are general and abstract, domain specific knowledge and memorization don't help. Figuring out the rules, the goal, the controls, and how to solve in a reasonable budget all indicate some level of general ability.