You know what's also faster to roughly get the amount of tokens? string.length/5
It is not helpful in actual tokenization, though.
alt Hacker News
It is not helpful in actual tokenization, though.