logoalt Hacker News

venusenvy47today at 1:55 PM1 replyview on HN

Where do you find this information for each model?


Replies

ComputerGurutoday at 2:06 PM

The tokenizers are included in the open s̶o̶u̶r̶c̶e̶ weights releases; you wouldn’t be able to use the weights without the corresponding encoder/decoder, in fact.