logoalt Hacker News

numpad0 • today at 7:41 PM • 0 replies • view on HN

Aren't tokenizers a hack to turn Latin alphabet texts into a pseudo Chinese with better per-unit information density?