logoalt Hacker News

Tiberium • yesterday at 6:18 PM • 1 reply • view on HN

There's also a tokenizer efficiency difference: modern Claude's 100K tokens are about ~60-65K modern GPT tokens, so in reality the Luna cutoff is much further away than the Haiku one.

You can test with Anthropic's count_tokens endpoint or with https://crates.io/crates/tokwc


Replies

AtNightWeCode • yesterday at 6:38 PM

> ...this tokenizer, the same input text produces approximately 30% more tokens on Claude Haiku 5.5 than on Claude Haiku 4.5.

So, it is might be even worse.

➕ show 1 reply