logoalt Hacker News

captainmuon • today at 7:49 AM • 1 reply • view on HN

Right, but my point was, I should be able to just send something like

    {'continue_conversation_id': 123, 'message': 'another message?'}
Seems like it would be even less work for the LLM host to just check if ID 123 is still in a cache (and if not, load it from a database) than decoding my request and checking cryptographic signatures. Right now the whole conversation history with blobs functions like a really long ID.

Replies

the_mitsuhiko • today at 8:16 AM

But then on a cache miss you need to resend the whole thing. Cached websocket connections on OpenAI work like this in a way. But once you lose the connection then you lose that.

Your idea though is the core of the responses API with store where everything is stored on their side and then you append to it.

For the LLM there is no difference btw. In either way you need to locate the cache and use it.