logoalt Hacker News

rererereferred • today at 2:13 PM • 2 replies • view on HN

I wonder if their very long words result in more or less token usage.


Replies

amunozo • today at 4:18 PM

Very long words in German are just compounds, made from individual words or morphemes. It is the same as in English if you remove spaces. Subtokens will be equivalent with or without examples.

thenthenthen • today at 3:31 PM

This is an interesting question, Chinese is way more compact in Character count vs English (about 40%?), let alone german, but yeah.. Chinese makes up for it with… total character count that runs into the thousands…