> showed that the LLMs reproduce the human behavior of overgeneralizing early and failing to, as the paper says, sufficiently explore the space[0].

Perhaps because there is only a real drawback to doing so if avoidance of bias is explicitly rewarded for some external reason? Like, by definition, if the groups are equal to each other, there's no loss from such exploitation (a larger candidate pool only helps if you have a working screening process, and a same-sized sample across the groups doesn't actually even confer the benefits of a larger candidate pool under the assumptions). Whereas if the observed clustering on a small sample isn't illusory, then ignoring it (or even actively going against it) would be clearly suboptimal. The probability of being actively misled by the clustering is necessarily less than the probability of being led correctly.

Going back to the example, of course bad FE units are less likely to overperform than good ones; that's what's bad about them. (But units can also be situationally good or bad for many reasons beyond their base stats and growth rates. And in FE we can typically directly observe that data and don't have to rely on anecdotes.) So the overperformance you saw was legitimate Bayesian evidence.