alt Hacker NewsAnthropic's tokenizer being 2x less efficient means they're essentially charging you for whitespace. premium whitespace, mind you — each space character gets its own attention head.
Anthropic's tokenizer being 2x less efficient means they're essentially charging you for whitespace. premium whitespace, mind you — each space character gets its own attention head.
I don't know if it's fair to say that a tokenizer is being less efficient if it generates more tokens per text. I think it's more fair to say the tokenizer is more nuanced. The question is whether the additional nuance permits better model output, which could justify the additional token cost in inference.