alt Hacker News> It's the same with human intelligence though.
No, this is not the same observation. In "basic LLM", the answer is not "confused" or "fail to understand", the answer is "inconsistent with the understanding mechanism". It is not that they "fail to understand while trying to understand", it is that there is not understanding mechanism at all.
Humans can have different level of intelligence, depending on the individuals, the subjects, even circumstantial situations (someone being tired, someone being distracted, or just bad luck). But they never make the same kind of mistakes I've observed with "basic LLM", where they do "non sequitur" that does not make sense at all but has all the characteristic of imitating something said by someone who understood.
I still even see it sometimes with Claude. It says logical stuff, and then suddenly something that does not make sense and it snaps me back to reality: none of this, including the correct things, are the result of understanding the underlying concepts, it is just that the correct things are more probable to generate, and that suddenly, a nonsensical happen to also be probable for a given configuration.
Humans don't make the exact same errors of LLMs of course. Humans are very different beings.
So you recognize that Claude is not a human.
Humans make mistakes as well "inconsistent with the understanding mechanism", but they have a very different form, and you are so used to the particular failure mode of humans, that you don't think about it.
But aliens visiting earth likely would find some aspects of human mind very peculiar!
Examples:
Humans learning algebra (or really anything like playing music, paddling a canoe, etc.) have to go through lots and lots of trivial basic mistakes, and only learn to avoid them through repetition and pattern matching on earlier experience, rather than relying on "reasoning".
A "pure reasonable being" would simply be learned the rules for algebra then go ahead and make perfect deductions applying the rules -- but humans are very clearly not such beings. Humans can know the rules for algebra perfectly well, then still go ahead and make mistakes until enough training has been done until we say you have "learned" it (be able to pattern match on previous experience).
Imagine humans being employed by aliens to do algebra, then aliens seeing humans basically do "2 + 2 = 5" (just on a higher complexity level). Like very human in first year in university WILL do with their formulas. What would you conclude about humans and their relation to "real understanding"?
Or another example: Humans engage a lot in post-rationalization, having first made up ones mind, then finding the reasons for the choice afterwards. (Most striking example of post-rationalization is the experiments on patients with severed brain half connections where one brain half invents a reason it can believe in for a choice made by the other brain half; https://en.wikipedia.org/wiki/Split-brain -- but if you look at pretty much any political issue for instance it is clear that people are driven at least as much by being herd animals as by doing any reasoning -- the majority of humans decide what people they belong with first, then figure out why afterwards).