logoalt Hacker News

paulryanrogers • today at 7:02 PM • 1 reply • view on HN

If tokens use UTF-8 bytes then it could be a factor.


Replies

ben_w • today at 7:17 PM

A token is an index into a dictionary. What the dictionary uses isn't seen, this is why they couldn't count the 'r's in 'strawberry'.