alt Hacker NewsIf 10,000 people guess 10,000 fair coin flips each one of them will get more guesses right than any of the others, one of them will get fewer guesses right than any of the others, and the gulf between the two is likely to be over 4 standard deviations wide. I'm certain that I, being an untutored schmuck from Pittsburgh and having thought of this almost immediately after reading about this contest, cannot be the first person to realize this is a potential problem for a forecasting contest. But I can't find anything they've done to mitigate that problem. Can anyone clue me in?
Replies
I don't understand your analogy. Are you just suggesting that luck plays too large a role in this contest? Clearly there is some "skill" or ability factor because AI's have been scoring higher and higher each year. Also, they make reference to superforecaster humans, who are presumably consistently better at forecasting than their peers.
They aren't guessing heads or tails, they give odds for each event. It's more like eyeballing a thousand coins to guess how fair they are, and then flipping each one just once.
Some are weighted to be 99% heads, others are 10% heads etc.
You could have 1,000,000 people guess random percentages for each coin, but suppose 10 of the coins are weighted 100% heads. To guess within 25% of the true value for all 10 of those coins would be roughly 1 in a million.
So a lucky guy guesses within 25% for all 10, he'd have another 990 coins he's being judged on.