alt Hacker NewsI think that switching from fixed token sets to something like H-Nets [0] (discussed on HN [1]) can at least improve on this issue, if not solve it completely.
Because, right now the main issue is that some languages are tokenized much more efficiently than others and it is a zero sum game, since there are a limited number of tokens and by adding tokens that are needed for one language you will inevitably have to remove tokens for another language and therefore decrease efficiency for it.
[0] https://www.cartesia.ai/blog/from-tokens-to-bytes-part-1
Exactly. Currently LLMs use a fixed tokenizer optimized for English and (to a lesser degree) Western languages. The H-Net architecture doesn't use a fixed tokenizer, it uses dynamicaly adaptive chunks of bytes, and (in a two-stage setup) chunks of chunks of bytes.
However, they (Cartesia) haven't yet published a technical paper on their new version of H-Net, and it is unclear if they will.