alt Hacker NewsFWIW: "Baba Is You" is 7 years old and heralded as one of the greatest puzzle games of all times, with guides and solutions shared all over the internet. How to beat this game is 100% in the training set.
Replies
modeless • today at 8:59 PM
It turns out knowledge and ability are not the same thing. We should test both.
It was our original assumption. Yet, we went through trajectories and agents did not recall solution. It is with a sharp contrast with task for which agents magically generate solution, e.g. https://openai.com/index/why-we-no-longer-evaluate-swe-bench....
In a few instances (we covered it in Caveats) Gemini 3.5 Flash "knew" which level it was, but misremembered, and went with a wrong solution.