alt Hacker NewsWhere's the human baseline for the benchmarks?
Also, you said:
> The best performing model scores a 66.8. So does this prove LLMs can “understand” the code? I would argue that the answer to this is “no”.
How did you reach that conclusion?
Where's the human baseline for the benchmarks?
Also, you said:
> The best performing model scores a 66.8. So does this prove LLMs can “understand” the code? I would argue that the answer to this is “no”.
How did you reach that conclusion?
I've considered setting up a human test as a starting point. But as a hobbyist that would require volunteers which I didn't think I'd be able to get for this project. Perhaps version 2 of the benchmark could do this if there is enough interest.
> How did you reach that conclusion?
Based on the fact that the task set so far includes fairly simple goals and none of the models scored 100%.