alt Hacker NewsI agree with you but I'd point out that unless you've read the book it's difficult to know if the answer you got was accurate or it just kinda made it up. In my experience it makes stuff up.
Like, it behaves as if any answer is better than no answer.
So do humans asked to answer tests. The appropriate thing is to compare to human performance at the same task.
At most of these comprehension tasks, AI is already superhuman (in part because Gary picked scaled tasks that humans are surprisingly bad at).