logoalt Hacker News

Generate fonts where every LLM token is the same width

39 points • by z-mach9 • yesterday at 12:30 AM • 10 comments • view on HN

Comments

croemer • today at 3:04 AM

I typed some German and it was breaking up words so much more than English. Not really surprising given tokenizers are optimized for most commonly used text.

Here's the token efficiency of a corpus translated into various languages and tokenized with the latest OpenAI one:

  Language              Relative tokens
  --------------------------------------
  English                    1.00x
  Portuguese                 1.23x
  Chinese (Simplified)       1.25x
  German                     1.31x
  Spanish                    1.32x
  French                     1.37x
  Arabic                     1.38x
  Chinese (Traditional)      1.42x
  Korean                     1.47x
  Swahili                    1.49x
  Hindi                      1.57x
  Japanese                   1.66x
  Burmese                    3.16x
  Amharic                    5.78x
  Santali                   13.70x
Source: "Tokenizer Fairness in 2026", a reproduction/extension of Petrov, La Malfa, Torr & Bibi, "Language Model Tokenizers Introduce Unfairness Between Languages" (NeurIPS 2023), using FLORES-200.

https://github.com/partyfly/tokenizer-fairness-2026

dTal • yesterday at 2:01 PM

This page reliably hangs Firefox 155 at 100% CPU for me.

➕ show 3 replies
nxtfari • yesterday at 2:25 PM

Haha this did help me generate empathy for the assistant. Neat idea.

LoganDark • yesterday at 12:59 PM

The kerning is absolutely awful in Safari. Looks fine in Chrome though...

kittikitti • yesterday at 7:13 PM

I wonder how this would look in Chinese Mandarin.

➕ show 1 reply