alt Hacker NewsOh, now I see where we have an actual difference of opinion. I don't think you can deny that even Finnegan's wake proceeds one token at a time; your interpretation of it may require more context or out-of-order interpretation, but that's just as true when observing text in German or Japanese, which have word ordering constraints that are alien to English speakers. How it was written is irrelevant; all we can observe is how it was presented. Of course we can observe each other's inner life, but we do so one token at a time, even if the process of producing each token is done (internally or actively) via a backtracking or zeitgeist approach.
You seem to believe, on a more fundamental level, that LLMs are simply not capable of producing text that has deeper connections to itself or represents abstract thoughts. In my opinion, 99% of text written by humans does not show this, just as 99% of text produced by LLMs does not show this, but both have the capability, and I don't believe that LLMs are constrained in such a way that they can never do this.
I am no linguist, but I believe this is referred to as surface structure and deep structure. What you describe is a line of text that is grammatically somewhat adequate to pass as readable and you treat all text the same. when we read the text, we decipher meaning out of according to word references and syntax - just like with Python or C++. However, if this were solely the case, we could not read Finnegan’s Wake. Probably vast majority of modern poetry would be unread by anyone, as would be pretty much all major works of philosophy Kant onwards. Deep structure is what according to Chomsky et al. gives meaning to the language, ie. somewhat logical structure behind the mere words. English word strict has order, need it but actually not does. You skibidi rizz swag grok brah also, barely. We use language in the extended meaning of the word to create a model of the world and somehow the past riverrun skibidi transmits that model to others. This is what I believe Bender also tries to say in their paper.
Now, you could claim that LLM’s have this deep structure, create models of the world and are basically just like us, and certainly many here are adament that this is the case, being aghast how someone can “insult” LLM’s by calling them parrots. However, there really is not much proof to back up this belief. Usually LLM’s seem to copy existing surface structure from whatever source, and when it deviates from these patterns, it usually becomes incomprehensible. There’s much hoopla about LLM’s solving hard maths, but it seems even there they are mostly generalising from vast amounts of training data, rather than actually reasoning: https://arxiv.org/pdf/2410.05229