alt Hacker NewsDoesn’t that blow your token cache hit rate and balloon your costs (essentially everything is billed at the input token rate, not the cached rate)?
Doesn’t that blow your token cache hit rate and balloon your costs (essentially everything is billed at the input token rate, not the cached rate)?
If it moved a block like
<system_prompt>
<tools_etc>
<project_prompts>
<user_call>
<llm_response>
=>
<system_prompt>
<tools_etc>
//Removed project_prompts
<user_call>
<llm_response>
<user_call>
<project_prompts> //reinserted
<llm_response>
Cache would break but if you did instead
<system_prompt>
<tools_etc>
<project_prompts>
<user_call>
<llm_response>
<user_call>
<project_prompts> //Duplicated with new user messsage
<llm_response>
It wouldn't bust cache, it would just make your input prompts slightly larger. Technically inefficient as you're duplicating the same rules over and over but I imagine for a smallish checklist/principles that it is tremendously more efficient than a cache break every message