alt Hacker NewsIt's one thing that the APIs don't return the reasoning tokens in a readable fashion. But then why do I have to send them the whole conversation every time? If they don't allow me to see parts of the conversation then they should have the burden of storing and transmitting it, and shouldn't charge more and more the longer the conversation gets (unless opaque token caching kicks in) like some LLM Shlemil the painter.
Replies
charcircuit • today at 6:19 AM
>and shouldn't charge more and more the longer the conversation gets
The cost of generating token N is O(N) with KV cache so it's unrealistic to expect to not be charged more the longer the conversation is if you are looking for the minimum price.
➕ show 1 reply
Because if you don't, you trash the KV cache and things get way more expensive. We wrote about it here: https://earendil.com/posts/prompt-caching/
Encrypted reasoning content is one of the things we keep as blobs in the transcript, even though we can only send it back to the original provider that created them, as otherwise we cannot keep caches live.