alt Hacker NewsAI simulates reasoning by lighting up the vector space (or concept space) weighted around a token so they it understands all adjacent words or concepts in that space.
This is a brillaint way to simulate reasoning, but its likely not how we reason ... simply how we store reasoning in writing.
Its useful if you know how to use it, its dangerous if you think its more than that.
But tl;dr it can (since its uncompressing our lingusticially stored reasoning from books) arrive at reasoning a DIFFERENT way than our brains did... and this isn't right or wrong.
Where it diverges is when it must move beyond the text or even the synthetic possible text of all vector spaces combined (aka novel territory) and it can't conjecture or test those outcomes well. But to be fair, neither can MOST humans.
Replies
Can you tell me your reasons for suspecting this likely isn't how we reason, or even a good analogy?
I'm not convinced—and certainly don't find it obvious—that this couldn't ultimately also be how we reason as humans.
It's clear that there's an enormous amount of leverage built into language-as-practiced that one can use to engage in a broad spectrum of reasoning, from the extremely fallible off-the-cuff conclusion to the deeply-considered and rigorous proof. How do we know this leverage is built into language-as-practiced? Because LLMs can do a broad swath of it.
But how do we know we're not doing something similar?
I don't think we can assume that we're not simply by observing that we're not digital and we don't use matrix multiplication. Why immediately dismiss the possibility that there might be a similar, but biomechanical, computation at play in our heads that plays in the same space of vectors?