alt Hacker NewsIs the author only running their model against one benchmark? I don't think anyone finds that difficult to achieve, the difficulty comes when you want to make the model not benchmaxxed to a specific benchmark, and generalize so it can solve problems not part of the training data, but seems this model is specifically for not this? How useful is that?
If you just wanted to pass these specific tasks in this specific benchmark, and wanted to do so cheaply, I'm sure a non-LLM-based approach would yield better results for even cheaper, since what the author's model does, seem to basically be "solve ARC puzzles", not a general LLM or "coding" LLM.
Replies
The whole point of his model is to optimize for a very specific benchmark.
BUT, he does not use labels when training, so the model does not know the answers.
[dead]
I read this as a response to the current hype around LLMs. He is showing computers can solve these issues, without using an LLM architecture. A lot of people have sort of forgot that machine learning is more than just LLMs these days.
I found it to be a very interesting angle.