But then on a cache miss you need to resend the whole thing. Cached websocket connections on OpenAI work like this in a way. But once you lose the connection then you lose that.

Your idea though is the core of the responses API with store where everything is stored on their side and then you append to it.

For the LLM there is no difference btw. In either way you need to locate the cache and use it.