My hypothesis of the mismatch is centered around "read" - I think that when you wrote it, and when others similarly think about that scenario, the surprise is because our version of "read" is the implied "read and internalized" or at bare minimum "read for comprehension" but as very best I can tell the LLM's version is "encoded tokens into vector space" and not "encoded into semantic graph"

I welcome the hair-splittery that is sure to follow about what it means to "understand" anything

The difficult part is proposing an experiment to check a new idea.

It is possible that such a human wouldn't come up with anything new, even if they could.