logoalt Hacker News

tcp_handshaker • today at 11:36 AM • 2 replies • view on HN

You can easily test this yourself with the SOTA models....or read the corroborating literature...

"General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks" https://arxiv.org/abs/2604.11778

"...General365, a benchmark specifically designed to assess general reasoning in LLMs. By restricting background knowledge to a K-12 level, General365 explicitly decouples reasoning from specialized expertise. The benchmark comprises 365 seed problems and 1,095 variant problems across eight categories, ensuring both high difficulty and diversity. Evaluations across 26 leading LLMs reveal that even the top-performing model achieves only 62.8% accuracy, in stark contrast to the near-perfect performances of LLMs in math and physics benchmarks..."

Replies

xnorswap • today at 11:58 AM

What proportion of the human population could answer the example from that paper?

    Question: Strangers A, B, C, D, and E line up from youngest on the left to oldest on the right. Their clothing
    colors and shoe colors all differ, and they come from five different regions.
    Known facts:
    1. A is from Morocco.
    2. D is five years older than B.
    3. E is older than A.
    4. C stands next to D.
    5. A stands next to B.
    6. The person in teal shoes is not adjacent to the person from Vanuatu.
    7. One twelve-year-old wears yellow shoes.
    8. The person in orange shoes wears white clothing.
    9. The person in blue clothing is from Chile.
    10. The youngest person wears red shoes.
    11. Counting from the right, the fourth person comes from South Africa.
    12. E wears yellow clothing.
    13. The person in green shoes does not wear multicolored clothing.
    14. Two people are twelve years old, ordered by birth month.
    15. One adult is thirty-five years old, and that age is sixteen less than the combined ages of the other four.
    If you multiply every possible age C might have, what number do you obtain?
What does "The person in green shoes does not wear multicolored clothing" even mean?

Nowhere is "multicoloured" defined, are we to assume it should be treated as a colour and implied that someone else must be wearing "multicoloured clothing"? Because strictly that doesn't logically follow, and it ought to be phrased as "The person in green shoes is not the person wearing multicoloured clothing" if that is the case.

This is an extremely hard logic puzzle, especially since it's revealed at the end that there are multiple solutions.

I'd expect anyone to struggle unless armed with prolog.

➕ show 1 reply
fn-mote • today at 12:41 PM

Super interesting, thanks for the reference.

I particularly found the note about “local collapse” helpful (near the end of section 3). The idea is that even though benchmarks contain a wide variety of different reasoning tasks, each individual problem requires only a few skills - unlike this benchmark where they deliberately construct tasks that span many categories.