alt Hacker NewsCost per token doesn't say a lot, but "Cost per benchmark task" is also meaningless if your task is difficult enough that the cheaper model has no chance of cracking it.
Replies
efromvt • today at 8:39 PM
Isn't the benchmark working exactly how it should in that case?
➕ show 1 reply
janalsncm • today at 8:28 PM
Similarly, tasks that are too easy also aren’t ideal either. If a small model makes mistakes and backtracks but eventually cracks it, it will be using a lot more tokens than a bigger model that does it all with minimal mistakes.
➕ show 1 reply
That's not meaningless at all, it's a great metric! If the "cost per correctly-solved benchmark task" is infinity, you know not to use the model.