logoalt Hacker News

kgeistyesterday at 1:30 PM0 repliesview on HN

The original tokens have Ġ instead of space. I had this issue too when writing an inference engine for Qwen. You have to "normalize" those special characters.