Smaller and/or heavily quantized models have a tendency to do this. The reason is because the auto-regressive nature of the llm dictates that its output should be its own input. You effectively purposefully create a feedback loop. That loop will manifest like this once the possible number of output states starts to converge on a smaller subset of the available tokens. A really interesting side effect of using speculative decoding is that you can detect that the llm is going to loop. You'll see this monotonic increase in the efficiency of the predictor and once that cross a certain threshold the loop is either already underway or it becomes inevitable. I've seen such loops range across 10K token and if not for the speculative decoding signal I would have never realized a loop was underway, but once you know it you can write a simple piece of software to detect the loop. The lower the 'temperature' at which you run the LLM the higher the chance of a loop (because you are constraining the number of branches that lead out from every token). The smaller the model the bigger the chance that it will loop. The more quantized a model is the bigger the chance that it will loop.
There are interesting parallels between LLMs and biological structures as well, for instance spontaneous oscillation in nervous systems lacking inputs but with a feedback loop in the topology.
Any kind of feedback system is riding a fine line between amplification/generation and runaway behavior resulting in an oscillating loop. Electronics, mechanical control systems, biological systems and LLMs all share the same property: some fraction of the output is coupled back to the input. With LLMs the fraction is 100%. It's a small miracle they don't loop more than they do!
alt Hacker News
Smaller and/or heavily quantized models have a tendency to do this. The reason is because the auto-regressive nature of the llm dictates that its output should be its own input. You effectively purposefully create a feedback loop. That loop will manifest like this once the possible number of output states starts to converge on a smaller subset of the available tokens. A really interesting side effect of using speculative decoding is that you can detect that the llm is going to loop. You'll see this monotonic increase in the efficiency of the predictor and once that cross a certain threshold the loop is either already underway or it becomes inevitable. I've seen such loops range across 10K token and if not for the speculative decoding signal I would have never realized a loop was underway, but once you know it you can write a simple piece of software to detect the loop. The lower the 'temperature' at which you run the LLM the higher the chance of a loop (because you are constraining the number of branches that lead out from every token). The smaller the model the bigger the chance that it will loop. The more quantized a model is the bigger the chance that it will loop.
There are interesting parallels between LLMs and biological structures as well, for instance spontaneous oscillation in nervous systems lacking inputs but with a feedback loop in the topology.
Any kind of feedback system is riding a fine line between amplification/generation and runaway behavior resulting in an oscillating loop. Electronics, mechanical control systems, biological systems and LLMs all share the same property: some fraction of the output is coupled back to the input. With LLMs the fraction is 100%. It's a small miracle they don't loop more than they do!