logoalt Hacker News

cubefox • today at 5:24 PM • 2 replies • view on HN

Exactly. Currently LLMs use a fixed tokenizer optimized for English and (to a lesser degree) Western languages. The H-Net architecture doesn't use a fixed tokenizer, it uses dynamicaly adaptive chunks of bytes, and (in a two-stage setup) chunks of chunks of bytes.

However, they (Cartesia) haven't yet published a technical paper on their new version of H-Net, and it is unclear if they will.


Replies

brainwad • today at 5:47 PM

It's not optimised deliberately for English, right? It's just that byte pair encoding naturally favours English and other Latin languages because they are dominant in the training data.

Even if you use bytes only, Unicode is biased towards English and other Latin languages, too.

➕ show 4 replies
stephantul • today at 6:15 PM

Non-latin languages also take more bytes due to utf-8 though

➕ show 3 replies