It doesn't loop though -- it has continuously updating context -- and if that context continues to head one direction it will eventually break down.

My own personal experience with LLMs is that after enough context they just become useless -- starting to make stupid mistakes that they successfully avoided earlier.