logoalt Hacker News

arjieyesterday at 4:38 PM1 replyview on HN

I just put the varying parameters in a trailer prompt and have them change every time. It doesn’t matter because the cache is prefix keyed. You lose caching for the last 20 tokens or so but that’s not a big deal. Moving it to a tool call makes it too slow (needs full roundtrip).

If you’re constructing the prompt you don’t have to jam everything together you can arrange it appropriately.


Replies

NeutralForestyesterday at 4:46 PM

Yes indeed! Mostly don't put changing data in the beginning or prepend.