A tokenizer is not a dictionary any more than an alphabet is a dictionary.
The Chinese alphabet is very much a dictionary. All the major tokenizers are far larger.
alt Hacker News
The Chinese alphabet is very much a dictionary. All the major tokenizers are far larger.