LLMs and Self-Referentiality

Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that aren’t true. RAG (I don’t mean embedding indices, but any information ingest such as an agent harness executing a search) are somewhat effective in covering for it, enough to make them very useful! But when it does go wrong, it’s generally the same reasons. It has a certain nature and sometimes you run afoul of it.

But I suppose it doesn’t harm its reasoning!

I don't think LLMs are properly self-referential. They reference a frozen training reality, which is not itself, but the old description of itself and the old world. Being aware would probably include continuously updating yourself (learning) from experience, including experience of oneself.

Was Hofstadter ever arguing that intelligence requires self-referentiality?

I haven't read his stuff in a long time, but from what I recall he was saying more that something about consciousness and the sense of self is based on self-referentiality. Not that intelligence requires self-referentiality.

I also remember having the sense that Hofstadter didn't really understand the "hard problem of consciousness". His discussions seemed to somehow confuse the sense of self with having subjective experience. But like I said, it's been a while since I read it.

It seems like it is the bitter lesson of the emergence of intelligence.

Kind of question-begging towards the idea that LLMs are self-referential in the same way that Hofstadter means. Even if they loop, they don’t do so strangely.

If self-reference and strange loops aren't intelligence, and emergence is merely a byproduct of universality, then what on earth is intelligence? What is this thing we question as intelligence, and what are the values that take shape from it?

Can we truly define intelligence as prediction, prediction as compression, and compression as the process of finding the upper bound of Kolmogorov complexity?

Can the statistical compression of data really explain everything? I don't know. What exactly is intelligence? Honestly, in everyday life, I rarely think about what intelligence actually is. I usually just focus on what the task at hand is and how to get it done, which makes this a fascinating question.

When you code with AI, you realize there is something fundamentally different from humans. The qualities that make a good senior programmer and the qualities that make a programmer good at orchestrating AI agents are similar, yet there is a subtly different feel to them. I might not be able to fully articulate it, but...

What exactly is the fundamental difference that creates this subtle distinction?

[flagged]