alt Hacker NewsEver heard of the infinite monkey theorem?
This is basically what LLMs do on really hard tasks. Prompt it a million times on a really hard problem and it might output the correct answer once.
Replies
artninja1988 • today at 4:04 PM
>Ever heard of the infinite monkey theorem?
Even if every atom in the universe were a supercomputer generating a trillion trillion random characters every second since the Big Bang, the chance of producing Hamlet would still be essentially zero.
The infinite monkey theorem assumes random distribution of symbols*.
Given the tokenizers have a vocabulary in the 10k-100k range, "a million attempts" will generally still only get the first token of the answer correct.
Even really rubbish models, e.g. talkie, the "what if we only use pre-1930s data to train a model?"** model, had to be almost all the way to the right answer to reach the really low HumanEval pass@100 score of ~0.04 (I'm only eyeballing the relevant chart).
* Actual monkeys not being like this is, while amusing, irrelevant
** https://talkie-lm.com/introducing-talkie