alt Hacker NewsCompletely the same as regular text, except punctuations are centered instead of staying at the bottom of the line. CJK Han likely encodes tokens to character one on one. Which brings an interesting question, are Chinese characters more efficient for NLP? In the sense that semantic meaning of a word is not chopped up into partial "tokens".
Perhaps relevant: Chinese language is not more efficient than English in vibe coding: A preliminary study on token cost and problem-solving rate https://arxiv.org/abs/2604.14210v1