alt Hacker NewsYou say that very confidently - but why shouldn't an LLM have learned a character-level understanding of tokens?
LLMs would perform very badly on tasks like checking documents for spelling errors, processing OCRed documents, pluralising, changing tenses and handling typos in messages from users if they didn't have a character-level understanding.
It's only folks who have absolutely no idea how LLMs work that would think this task presents any difficulty whatsoever for a PhD-level superintelligence :)
Replies
fzzzy • 01/20/2025
The llm has absolutely no way of knowing which characters are in which token.
LLMs are fed token ids, out of a tokenizer.... no characters. They don't even have any concept of a character.
You are in a discussion where you are just miles out of your depth. Go read LLMs 101 somewhere.