"Even SOTA models at the end of their context limit behave REALLY illogical "

The trick is not go to that limit, but stay under 50% or even better 25% of context length. But backups are a smart thing anyway.