alt Hacker News> Increases in LLM scores are now mainly driven by post training (evidence in next section) and are probably a function of amount of synthetic data. They are learning to solve ARC tasks, not learn general abstract reasoning
Agreed and that's for any benchmark. Private tests are better but you still have to trust the provider to not log and use them for training.
That's why I like when a new set of tests like a new ARC-AGI version is published, that's where you can see which of the models abstracted to more general capabilities instead of being focused on the previous tasks. Most models completely fail new ARC-AGI tests.
The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. You hit a ceiling very fast and investing into more compute will give you diminishing results. So yes, you can train a custom model to do somewhat decently on a specific set of tasks but then what?
Replies
> The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results.
I think thats unfair. Perf-compute is often logarithmic and will always saturate . Reaching the plateau faster is valuable as it often leads to better peaks (held true here and also look at modded nanogpt)
And more compute increases the perf (after dealing with other scaling problems)
Then nothing - that's awesome. People think that LLMs are the know-all do-all solution to every problem now.
Putting solutions in terms of cents is a great way to potentially win over some ai boosters imo. There are other ways to solve hard problems.