alt Hacker NewsSo like- I considered this idea of using a fast decision model to prune before compaction, but I scrapped it because A; im not bottlenecked by speed, and B; cache invalidation dark magic
From my understanding, if we have turns [A, B, C, D, E] and we decide that turn C is useless context, then we lose our cache for the [D, E] segment. (plz correct me if im wrong, I havent tested this)
Personally, I pay no mind to session state nowadays. I exclusively use provider native hosted compaction, which if you believe the providers, is the way to go. For context management- I spend most of my effort on the things which dump output back to the model.
If anyone has any good reads on benchmarking OpenAI and Anthropic hosted/native compaction against custom stuff, I would love to read!
Replies
It's because the cache comes from the prefix. `f(f(f(A,B),C),D)` is quite different from `f(f(A,B),D)`.
I use dynamic context compaction in open code plugin. Then modified it further to include eviction.
Since I run everything local, speed benefits from keeping context low ~128k. This gets me through a single agent on multiple tasks through the same repo. It seems to break around 6M but that's more probablistic.
KV cache is definitely valuable but context poisoning happens regardless of size and it's probability increases regardless of intelligence. So at some point you're going to suffer from context overload regardless.
Yeah, I also think decision models for pruning context isn't worthwhile. You're paying to load it into context (of your decision model), in order to save on loading the same data into context (of your generation model). The only way the economics makes sense is if you need a much smarter model for the compaction, but can't pay for it to load all of that context (because it's uncached or something). That sounds like a niche but it really doesn't exist. Cheap models are good enough, and smart models are cheap enough.
Personally, I use a harness extension that auto-compacts my context just before the cache expires. Combined with the default compact-when-full, this ends up being pretty much always what I want. It's one less thing to think about.