alt Hacker NewsFrench text is somewhere around 10-30% longer than the corresponding English text. I would guess much of what you save on smaller vocabulary is lost on the lengthened text.
French text is somewhere around 10-30% longer than the corresponding English text. I would guess much of what you save on smaller vocabulary is lost on the lengthened text.
What would matter more is the token length, depends how well your tokenizer was trained on french I guess.