And you know what the tokenizer is made of?
Weights.
A tokenizer is a deterministic string-matching program, it's not made out of weights in the same sense as a neural network itself.
alt Hacker News
A tokenizer is a deterministic string-matching program, it's not made out of weights in the same sense as a neural network itself.