alt Hacker NewsTokenizers define the alphabet on which the language model is trained. I don't want people to get the impression it's a module which can be swapped out or modified on its own. Alphabet size is a design consideration related to correctly encoding the training data.
Replies
semiquaver • today at 6:27 PM
I’ve been struggling to understand the reason for the newer apparently less efficient Anthropic token encoding. If all inputs are less efficient in this encoding, why does it exist? Has Anthropic released any information that would convincingly show it was anything other than a stealth price hike? Please don’t respond if you are speculating.
➕ show 2 replies
That's true, but it makes it difficult to compare pricing when it's based on tokens. Maybe we need a benchmark for price per a specific input, like enwiki8.