logoalt Hacker News

kingstnap • today at 5:51 PM • 0 replies • view on HN

Beating even a very small LLM at next token prediction is basically impossible.

[2212.11281v2] Language models are better than humans at next-token prediction https://arxiv.org/abs/2212.11281v2

We knew this many years ago. Like GPT 1 era.

Actually you can play it here. The game is pretty frustrating ngl. And I can clearly see why induction heads are a thing. Because when a word repeats you guess is to just guess the same thing from before.

Language modelling game https://rr-lm-game.herokuapp.com/