alt Hacker NewsI have a stupid benchmark: Bongard Problem 2 from here (https://matheminutes.blogspot.com/2019/04/bongard-eggs.html). It's an open-ended visual task that is easy enough for humans, but models are still somewhat bad at visuals, so it's one of the few remaining reliable ways to trigger hallucinations. This one answered correctly, but then felt the need to add a fun fact that was wrong:
> The rule separating the two baskets is layering (which color is on top). Left basket: In every pattern on all six eggs, the blue shape is on top of (overlapping) the green shape. Right basket: In every pattern on all six eggs, the green shape is on top of (overlapping) the blue shape. [true solution]
> (Note: Both baskets contain the exact same distribution of motif counts per egg—two eggs with 2 pairs, three eggs with 3 pairs, and one egg with 4 pairs—serving as a classic Bongard distractor to hide the occlusion rule.) [factually incorrect and also would make no sense as a distractor]
And then trying a harder puzzle just makes it bullshit completely.
I don't understand why Gemini models always add these "it's typical XYZ" bits, which are typically so much worse than everything else in the answer.
Replies
If you have an Ultra subscription (I don't or I would try it) I would be really curious if Carbon is able to solve your harder Bongard problem without making stuff up given it's absurdly high AA Omniscience non-hallucination score, it's supposed to be like 30%+ better than nearest non-Google competitor. Although Hydrogen xhigh apparently scored within 5% of Carbon so who knows ¯\_(ツ)_/¯
The sample problem I kind of gave up on and guessed sort of... it couldn't be sorted by color that's too obvious.
And then the real problem number 1. The answer is "Left - each figure contains an even number of arrow heads. Right - each figure contains an odd number of arrow heads"... but on the right I can clearly see that the three eggs across the top have an even number of arrowheads.