logoalt Hacker News

dom96 • yesterday at 10:12 PM • 1 reply • view on HN

I've considered setting up a human test as a starting point. But as a hobbyist that would require volunteers which I didn't think I'd be able to get for this project. Perhaps version 2 of the benchmark could do this if there is enough interest.

> How did you reach that conclusion?

Based on the fact that the task set so far includes fairly simple goals and none of the models scored 100%.


Replies

StilesCrisis • yesterday at 11:41 PM

How much time did you give them? At maximum effort I can't believe they just threw up their hands and said it was too hard. I'm guessing you really mean they failed a zero-shot?

➕ show 1 reply