Actually you can play it here. The game is pretty frustrating ngl. And I can clearly see why induction heads are a thing. Because when a word repeats you guess is to just guess the same thing from before.
Probably high because humans aren't trained on lowering cross entropy loss. Kind of funny to think that next token prediction is actually an emergent capability of the human brain...
Beating even a very small LLM at next token prediction is basically impossible.
[2212.11281v2] Language models are better than humans at next-token prediction https://arxiv.org/abs/2212.11281v2
We knew this many years ago. Like GPT 1 era.
Actually you can play it here. The game is pretty frustrating ngl. And I can clearly see why induction heads are a thing. Because when a word repeats you guess is to just guess the same thing from before.
Language modelling game https://rr-lm-game.herokuapp.com/