logoalt Hacker News

marcosdumayyesterday at 11:24 PM0 repliesview on HN

The most important feature of a tokenizer is dividing the inputs into independent values that the neural network can work with. It's not the size.