alt Hacker NewsHow is solving unsolved problems a good eval? Once a problem is solved and released, you can’t evaluate a future model’s ability to solve it.
Replies
jsrozner • today at 6:14 AM
I think this is right - it can be seen as trying to get free feedback from the community. In that way it’s reasonably described as exploitative, since it’s not a good faith effort. The problem of ai slop being submitted to conferences to get publication counts is similar.
As I said, a proper eval is better, but it measures something real that gives a good training signal, and there is lack of good alternatives for math eval. Since the result is 372/4000, it is also unsaturated.