Completely the same as regular text, except punctuations are centered instead of staying at the bottom of the line. CJK Han likely encodes tokens to character one on one. Which brings an interesting question, are Chinese characters more efficient for NLP? In the sense that semantic meaning of a word is not chopped up into partial "tokens".
alt Hacker News
Completely the same as regular text, except punctuations are centered instead of staying at the bottom of the line. CJK Han likely encodes tokens to character one on one. Which brings an interesting question, are Chinese characters more efficient for NLP? In the sense that semantic meaning of a word is not chopped up into partial "tokens".