Beating even a very small LLM at next token prediction is basically impossible.
[2212.11281v2] Language models are better than humans at next-token prediction https://arxiv.org/abs/2212.11281v2
We knew this many years ago. Like GPT 1 era.
Actually you can play it here. The game is pretty frustrating ngl. And I can clearly see why induction heads are a thing. Because when a word repeats you guess is to just guess the same thing from before.
Language modelling game https://rr-lm-game.herokuapp.com/