alt Hacker NewsYann LeCun was saying 3 years ago that because token generation is auto-regressive, its mathematically impossible to generate a long stream of coherent tokens, because errors amplify exponentially.
and then models learned that they can back track and error correct
so much for "mathematically impossible..."
Replies
You insinuate here AI "learned".
I doubt that this was AI self-improvement.
I think it was largely the introduction of tool calling that allowed models to mitigate the issue of errors amplifying exponentially since it allows the model to understand if what it generated is correct or has issues that it needs to address. This addresses the potential lack of or low quality of world model by being able to reference the current state of the world.
> and then models learned that they can back track and error correct
You mean "Human developers learned ...", yes? Or was there really an all AI-driven, self-improving aspect to this?
[dead]
Also, almost any argument against LLM intelligence also applies to humans.
I very commonly see someone make some small mistake and end up going in the wrong direction, “accumulating stupid” as they go, sometimes for years.
Stop attacking Yann. I would say like 90% of the HN crowd was parroting Yann too.