logoalt Hacker News

cainxinthtoday at 1:36 PM1 replyview on HN

Is it fair to say that the “Rs in strawberry problem” will not be “cleanly” solved unless we advance beyond tokenization?


Replies

idiotsecanttoday at 1:58 PM

I think tokenization is probably not going anywhere, but higher layers need the ability to inspect 'raw' data on demand. You don't spell out most words as you read them, but you can bring the focus of your entire mind to the spelling of the word strawberry if you so choose. Models need that ability as well.

show 1 reply