alt Hacker NewsI think it's obvious that LLMs will be able to do "reasoning" far better than humans. We must separate our notion of what is remarkably human. Rarely is it the reasoning, it's the intuition that a logical path exists -- for example a mathematical proof that draws from separate sub-disciplines of mathematics, etc.
Consider that in a LLM, language inputs are tokenized and fed as inputs into the neural network, and connections in the network create output sequences that are not just syntactically correct (trivial) or form semantically plausible sentences (early transformers did this). LLM output sequences follow the deep patterns of language which include sometjhing that resembles reasoning as the model has learnt from its training data.
LLMs seem to fall short because they often fail at truly abstract reasoning tasks that humans find easy. If trained properly, LLMs can develop advanced representations of logical systems that will surely outpace what humans can do in terms of raw reasoning.
However, human mathematicians have not even unified around constructive mathematics as a must for the study of mathematics. This reveals that even highly evolved mathematical disciplines rely on objects whose characteristics do not lend themselves to full logical scrutiny and are in a way socially constructed and effectively hard to audit.
While notation in mathematics is incredible technology it is also a highly limiting factor that suffers major tradeoffs. Humans struggle to invent new notation fast enough and to discard outdated notation fast enough. If we do see an AI-powered boom in mathematics, I suspect our notion of notation and the fluidity we demand from it will change dramatically.
Replies
> If trained properly, LLMs can develop advanced representations of logical systems that will surely outpace what humans can do in terms of raw reasoning.
We have already trained the LLMs on most of the human knowledge base (so like 4-5000 years?) - imo training data will become a problem and will soon be more expensive than compute. Sure, you can work around some of this using synthetic training data but I personally would not count on general-purpose LLMs (especially LLMs aka transformer models) developing super-human representations of logical systems anytime soon.
I don’t see how it’s obvious that LLM’s will be capable of any mathematical, “reasoning.
LLM’s can infer relationships and maintain longer context chains in order to generate their output… it still happens that some times the output is correct depending on the training data, layers, context, etc. And it can get more accurate when we change the parameters of the model. But the algorithm isn’t “doing” anything here. It will generate something regardless of what it’s prompted with.
Maybe it’s right. But the algorithm is an algorithm. It doesn’t care what truth is. It’s generating BS essentially.
A human is doing a lot more work when performing mathematics.
It may be that LLM’s can be a useful tool in mathematical reasoning but it’s not obvious that it will ever be capable of it without a human, let alone be better than a human.
This argument is centered around the belief that language and reasoning flow bidirectionally- language can be understood first (we are here), and reasoning is the next natural rung of the latter (your thesis believes we will get here with LLMs).
I see language more as a medium for transcribing reasoning. While language certainly communicates reasoning, you can have reasoning without language, but not language without reasoning.
This paper seems to imply that current LLM's are just copying the training dataset's reasoning communication, not understand the actual reasoning. I don't think LLM's moving past this is "obvious" or even close to being inevitable.
> Instead, LLMs likely perform a form of probabilistic pattern-matching and searching to find closest seen data during training without proper understanding of concepts. While this process goes beyond naive memorization of words and the models are capable of searching and matching more abstract reasoning steps, it still falls short of true formal reasoning.