logoalt Hacker News

cortesoftlast Thursday at 6:44 PM3 repliesview on HN

How should they fix it?


Replies

s0ck_r4wlast Thursday at 7:17 PM

Make the cache TTL more adaptive. Have more tiers than just 5m and 1h. Long-running subagents become increasingly ubiquitous. There's no reason for Anthropic to not do a better job for this use case.

addaonlast Thursday at 7:53 PM

There are a lot of workflows where cache is likely to be consumed exactly once (e.g. anything chat-like where a single thread is extended; if a prefix cache hit is found, and a set of new longer prefixes is inserted in the cache, it is unlikely that the original shorter prefix will be consumed in later turns). I could see allowing each session key a small number (1?) of 24-hour cache entries, where inserting a new cache entry (perhaps the maximum-length prefix, perhaps an API-tagged specified-length prefix) consumes that slot and either boots the rest of the items from cache, or demotes them to 5 minutes or something. Basically: workflow awareness, not uniform handling.

randomblock1last Thursday at 10:18 PM

Default to 1h. Allow setting it to longer or shorter, granularly. Add /pause to mark it for eviction, for a token refund.