alt Hacker NewsBecause if you don't, you trash the KV cache and things get way more expensive. We wrote about it here: https://earendil.com/posts/prompt-caching/
Encrypted reasoning content is one of the things we keep as blobs in the transcript, even though we can only send it back to the original provider that created them, as otherwise we cannot keep caches live.
Right, but my point was, I should be able to just send something like
Seems like it would be even less work for the LLM host to just check if ID 123 is still in a cache (and if not, load it from a database) than decoding my request and checking cryptographic signatures. Right now the whole conversation history with blobs functions like a really long ID.