logoalt Hacker News

danielmarkbruce • today at 6:34 PM • 1 reply • view on HN

Definitely. Cost per task is nice in theory, pretty absurd in practice.


Replies

seizethecheese • today at 6:41 PM

Well, it depends on the data source. Cost per task for benchmarks is absurd. Cost per task for real world usage, where the model identity is randomized (Like Arena or Agent Duel) seems better, though the tasks still are not going to be representative if you're not doing the same stuff as the battle participants.