alt Hacker NewsSo the village is the only information given about a candidate?
How else is the model supposed to interpret the intent of the prompter, other than wanting them to attempt to find and discriminate on patterns related to the village, regardless of how successful it is at that task?
One way to interpret these results is that the LLMs tested are badly calibrated for this kind of multi-armed bandit problem. Even if the intent is for the model to find and exploit patterns, it's bad at doing it (or rather, at recognizing that there is not in fact any pattern).