logoalt Hacker News

ComputerGurutoday at 2:06 PM0 repliesview on HN

The tokenizers are included in the open s̶o̶u̶r̶c̶e̶ weights releases; you wouldn’t be able to use the weights without the corresponding encoder/decoder, in fact.