if you are refreshing a cache of 120k tokens, you have to input the same 120k tokens. you can't bounce the cache with 1 byte. A proper cache hit requires your entire context that is cached. if you change 1 byte in it, everything after that byte is a cache write and no longer a cache hit
alt Hacker News
if you are refreshing a cache of 120k tokens, you have to input the same 120k tokens. you can't bounce the cache with 1 byte. A proper cache hit requires your entire context that is cached. if you change 1 byte in it, everything after that byte is a cache write and no longer a cache hit