logoalt Hacker News

flakinesslast Friday at 6:23 PM1 replyview on HN

Oh it's good old tokenization vs for-LLM tokenizations like sentence piece or tiktoken. We shouldn't forget there are non-ML simple things like this one which doesn't ask you to buy more GPUs.


Replies

jamesgresqllast Friday at 6:49 PM

Haha, I like “good old tokenization”