Except, if some of the answered tokens are opaque to me, but I pay for them, the operator has no incentive to keep their number low [1]. For all I know the model could be spinning thumbs while thinking. If they cannot provide transparency, I would prefer that they not charge me for that part. Of course, they would prefer to charge me, so sure it is unrealistic to expect that they don't. I'm not trying to predict what they are going to do as a market participant, just stating my moral preference :-).

Also, isn't it without caching even something like O(N^2) because you have to replay the whole conversation on every request to reach the same internal state? My point was I shouldn't have to pay for cache misses if hitting the cache is not deterministic. Give me a guarantee that you keep the session "hot" for N minutes, and cached on disk for M months, charge a little bit more on average, but then the pricing is at least transparent.

[1] Except the general market pressure to keep total cost for the same problem solved lower than the competition.