logoalt Hacker News

Chu4eenoyesterday at 11:55 PM0 repliesview on HN

No, that's not how anything works.

You can finetune an LLM to a new tokenizer by nudging just a few layers (even wildly different kinds of tokens), and there's nothing stopping a lab from using someone else's tokenizer.