I've run into what I think might be an even bigger problem with the "just keep reading/outputting the next token" design: there's no going backwards.

This means that LLMs don't do well with input where text that comes later in the input alters how text that came earlier should be interpreted. Which is really quite common in practice.

Humans like to do things where you start with a dump of information, and then ask questions about it afterward. If a human reader didn't remember something pertinent to the question, they can go back and re-read now that they know what they're looking for. An LLM can't do that by itself. You can paper over that limitation with hacks, but they're hacky and unreliable.

Humans also correct misunderstandings and resolve ambiguities after the fact. LLMs have a really hard time with this. Once one way of interpreting some text has already made it into the LLM's context vectors, it can be surprisingly difficult for subsequent text to significantly alter that interpretation.

It's these sorts of aggravating failure modes stemming from a fundamental feature of the algorithm that seem to always produce the next AI winter. Because they tend to mean the same thing: research has gone down (and then, quote often, massively over-invested in) yet another blind alley that doesn't put us on a short easy path to AGI any more than the previous 75 years' worth of breakthroughs have. And getting out of it is going to involve not only fixing that one problem, but also a huge reset on some substantial portion of whatever tools and techniques were built up to paper over the previous algorithm's limitations in an effort to generate positive ROI on all that massive over-investment.