logoalt Hacker News

stephantul • today at 6:15 PM • 3 replies • view on HN

Non-latin languages also take more bytes due to utf-8 though


Replies

ben_w • today at 6:19 PM

Are bytes relevant? These things are billed by the megatoken, not the megabyte.

numpad0 • today at 7:23 PM

bit configurations aren't uniform across those bytes, everything should compress to similar sizes

guipsp • today at 6:26 PM

utf8 is just one of the encodings that exist, and is kind of irrelevant at the point it contacts the llm