The token usage is really interesting. I would imagine the most efficient thing is to keep the state of everything persisted, and past the cache expiration window, to automatically start a new session with the previously persisted state instead of just a long running conversation.

If someone solves this part of continual effective compaction + selective resetting at cache expiry, they're going to make a ton of money. Right now, only the token insensitive can use these sweet features.