alt Hacker NewsThat table assumes cache hit rate of 95% or better. Am I understanding this correctly that people really are doing such repetitive prompts (compared to each other, across the concurrent user base at that time) that only 5% or less need actually be computed by the intended LLM?
That is shocking. Is it per-token I wonder?
Replies
workbreak • today at 12:43 PM
Every tool call is essentially entire prompt so far sent again with the response and that's why cache rates are so high for agentic workloads. This really bites when using expensive models since most models are 1/10 for cached input.
_aavaa_ • today at 1:06 PM
If you are using their coding plan for coding, then yes you can easily hit such cache rates, with a good harness.
I’m getting 97%.
We have been running a lot of agentic benchmarks with the various loops and tool calls on longer threads - we routinely see 90%+
Just checking now: recent runs tau3[1] was at 96% and toolathlon[2] was at 90%
[1] https://www.induction.ai/docs/benchmarks/tau3 [2] https://www.induction.ai/docs/benchmarks/toolathlon