logoalt Hacker News

seizethecheese • today at 6:41 PM • 0 replies • view on HN

Well, it depends on the data source. Cost per task for benchmarks is absurd. Cost per task for real world usage, where the model identity is randomized (Like Arena or Agent Duel) seems better, though the tasks still are not going to be representative if you're not doing the same stuff as the battle participants.