logoalt Hacker News

semiquaveryesterday at 7:03 PM1 replyview on HN

Are there any advantages of the new tokenizer? Does it have a larger or smaller vocabulary or just differently weighted?


Replies

ianberdinyesterday at 7:35 PM

Well, in my view, it's just the most ordinary manipulation to avoid creating unrest. There is most likely no improvement inside.

Of course, these are my guesses, but did anyone feel the difference in the transition from Opus 4.5 to 4.6? In my opinion, no. And it's unlikely to be a matter of the tokenizer.