alt Hacker News>From a domain point of view, some are skeptical that bytes are adequate for modelling natural language
If I remember correctly, GPT3.5's tokenizer treated Cyrillic as individual characters, and GPT3.5 was pretty good at Russian.
>From a domain point of view, some are skeptical that bytes are adequate for modelling natural language
If I remember correctly, GPT3.5's tokenizer treated Cyrillic as individual characters, and GPT3.5 was pretty good at Russian.
I wonder if they treat each letter as a Unicode code point, and each of those is a token? I could see the same being true of other languages.