alt Hacker NewsSo, worse? Because we benchmark off token use when talking about token use, and everyone else understood that.
Replies
jascha_eng • yesterday at 9:43 PM
I mean it might lead to better performance on the model side. So the tokenizer is better but more expensive.
otikik • yesterday at 7:53 PM
It’s better for them
The most important feature of a tokenizer is dividing the inputs into independent values that the neural network can work with. It's not the size.