logoalt Hacker News

robrenaud • today at 3:05 PM • 0 replies • view on HN

Use different tokenizers for different languages. Or at least, use different tokenizers for different scripts.