alt Hacker NewsJust start pricing in bytes input/output. This whole "token" and "tokenizer" thing is an implementation detail that shouldn't even be leaking out into the API.
Providers change tokenizers all the time with model updates, and it's often not even possible to query/figure out how text is tokenized without actually just sending the LLM a request.
Just switch to charging for bytes of intelligence. Please. Claude Shannon figured this out decades ago.
That does not help you compare pricing between models since they can vary by (even internal, thinking) output size.
The best way to measure is really the end-2-end cost, price per task.