logoalt Hacker News

michaelmiortoday at 8:25 PM0 repliesview on HN

This depends on what your workflow is. There are use cases for tokenization that don't always involve immediately feeding the text into a model.