alt Hacker NewsYes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.
Replies
Stevvo • today at 8:17 AM
If you stop and think about the problem it really is quite simple. Just need to build a graph of the game state and then run A* to get to the end.
➕ show 1 reply
No but the session traces are available. It passes the sniff test considering how AGI-3 is scored and how this wrapper works.
For example on bp35 it took fable 290M and >12k simulated turns for 566 real turns and finish more efficiently than a human.
Regardless of the true score I think the takeaway is the benchmark measures the wrapper rather than the model.
https://huggingface.co/schema-harness