alt Hacker NewsThat's just general context rot, and the models do all sorts of off the rails behavior when the context is getting too unwieldy.
The whole breakthrough with LLM's, attention, is the ability to connect the "not" with the words it is negating.
Replies
noosphr • yesterday at 5:57 PM
Large enough is usually between 5 to 10% of the advertised context.
This doesn't mean there's no subtle accuracy drop on negations. Negations are inherently hard for both humans and LLMs because they expand the space of possible answers, this is a pretty well studied phenomenon. All these little effects manifest themselves when the model is already overwhelmed by the context complexity, they won't clearly appear on trivial prompts well within model's capacity.