alt Hacker NewsWhy do you need an API key to tokenize the text? Isn't it supposed to be a cheap step that everything else in the model relies on?
Replies
kouteiheika • today at 4:49 AM
I'd guess it's because they don't want people to reverse engineer it.
Note that they're the only provider which doesn't make their tokenizer available offline as a library (i.e. the only provider whose tokenizer is secret).
simonw • today at 4:16 AM
I'd love it if that API (which I do not believe Anthropic charge anything for) worked without an API key.
To prevent abuse? It's a completely free endpoint so I don't understand your complaint.