alt Hacker NewsBut there must be a ton of generic questions that people ask. Stuff like "What's the capital of country X?" - it's probably at least 10% of queries. Memories, custom instructions etc would invalidate them, but if you can return the answers basically free it's probably worth it.
Replies
imtringued • today at 8:19 AM
How is that cheaper? You now need to have a database of millions of possibly gigabyte sized rows. Also, transformers have quadratic complexity, so short queries cost practically nothing.
The only optimization that makes sense is per user prefix caching, because you are often sending the same system prompt over and over again or are continuing a conversation.
Questions like that cost a tiny fraction of a cent. "What's the capital of Sri Lanka?" cost a fifth of a cent at GPT 5.5 API price, and would cost a fraction of that if the question were routed to a more suitable, cheaper model. The output was 78 tokens.
By contrast, when coding, devs typically have hundreds of thousands of tokens in the context window, and may use many millions of input tokens per day.
Caching requires the full prefix to match exactly. If a single word differs near the beginning of the prompt, nothing after that can share the cache. So this type of caching would save a few queries that cost virtually nothing, but wouldn't help with the stuff where cost matters.