> I can think of a few good myths for today’s “AI”. Searle’s Chinese room comes to mind, as does Chalmers’ philosophical zombie. Peter Watts’ Blindsight draws on these concepts to ask what happens when humans come into contact with unconscious intelligence—I think the closest analogue for LLM behavior might be Blindsight’s Rorschach.

LLM's remind me of sprites, pixies, and the like, who are situationally helpful but require constant supervision. We're like modern magicians who learned how to summon these sorts of spirits and bind them -- imperfectly -- to our will. But their perception of truth and reality is "through the looking glass" relative to our own. They aren't lying, from their own frame of reference, even though what they say is untrue relative to ours.