I've considered setting up a human test as a starting point. But as a hobbyist that would require volunteers which I didn't think I'd be able to get for this project. Perhaps version 2 of the benchmark could do this if there is enough interest.
> How did you reach that conclusion?
Based on the fact that the task set so far includes fairly simple goals and none of the models scored 100%.
How much time did you give them? At maximum effort I can't believe they just threw up their hands and said it was too hard. I'm guessing you really mean they failed a zero-shot?