alt Hacker NewsThat's pretty innovative. But a brand new tokenizer and training method need a lot of effort to do the retraining. Whether it will be widely implemented depends on real frontier performance it can reach.
That's pretty innovative. But a brand new tokenizer and training method need a lot of effort to do the retraining. Whether it will be widely implemented depends on real frontier performance it can reach.
Exactly. I used to think new tokenizers for domain-specific models was the way (makes intuitive sense, better representations etc), but the added overhead of essentially training from scratch made it much more expensive than a FT.