> The model definitely remembers previous exchanges within the same conversation.

No it doesn't. They get added to its context, and it reads them afresh when answering the next question. That's not remembering.

If your short-term memory completely malfunctioned one day, so you had no ability to remember what was said to you a minute ago, then you would have to find workarounds. For example, you could write down everything someone says to you, then read your notes of the previous exchanges in that conversation in order to continue the conversation. That would be a good way to work around the fact that your short-term memory was broken. And if your notes were invisible to other people and you could read them really fast, then you could even make most people believe that you remembered what they said a minute ago. But you don't actually have a working memory, you're just writing down what they said and re-reading it while coming up with your next response.

That's exactly what LLMs do. That's not memory.

Not the model though. The model really only takes input text and produces output text. Memory within a conversation is achieved by the harness adding the conversation (or parts of it) to the input text. The LLM itself has no memory, it’s the augmented system of several orchestrated LLM calls that does.

Right, but that's still external to the LLM, it's just a KV cache that's stored on the provider side for performance reasons, so that the client doesn't have to re-send the whole chat history with every subsequent call in the conversation.

It still generates every response using the model's pristine state with every new API call; whether the context is provided from the client or from a colocated cache server doesn't really change that.

> The model definitely remembers previous exchanges within the same conversation.

Christ HN isn't what it used to be