alt Hacker NewsRich Sutton on AI creativity and discovery
Comments
The most successful applications like coding are not the result of pure LLM/generative modeling. They come from closing the loop with an agentic harness. The generate-test-selectively refine loop is the core modality of scientific work. An LLM + RL with Verifiable Rewards + feedback from compiler/terminal runs mimics this process to a great extend.
This is Fisher/Box feedback loop (https://www-sop.inria.fr/members/Ian.Jermyn/philosophy/writi...) implemented on a modern computational system. LLM is just a component. I wish Sutton had commented on this fuller picture of what we have now instead of commenting just on the LLM/Backprop side of things. I am honestly curious of whether such a loop can at least partially automate discovery.
There are more elements to discovery though. It is still not clear where the initial working model/hypothesis comes from or how the updates are selected (unless it is just parameter induction). I recently read about Hanson's Patterns of Discovery which aims in that direction. I have still not read it, but I am curious if it has any mechanistic clues.
"So that is my call to arms. If we want the full power of AI scientists, then we should share the goals with them so they can create, evaluate, discover, and in these ways fully participate in achieving the goals. Let’s be bold! Let’s fully automate Creativity and Discovery!"
Should we automate exercise and play as well? How about learning?
The machine didn't have a soul, so we donated ours.
Eureka! My AI found it!
The world will not be satisfied until we have read, and discussed every half-famous person’s opinion on AI.
Still about ten million discussions to go.
I think a lot of deep learning is compositional generalization. Models learn reusable pieces (abstractions, styles, procedures, constraints, etc) and recombine them in ways that may never have appeared as a whole in the training data. So even if the ingredients come from past data the final composition can still be novel in a meaningful sense
> That is, I would say that creativity requires that the new things generated be Evaluated. Without evaluation, and retention of the best, there is nothing created. The novelty flickers into existence but, if its value is unrecognized, it flickers away and is lost.
I really like the way he frames this here. I think a lot of people in the twitter comments (and maybe a few here) aren't reading past the introduction. He isn't saying AI systems are incapable of creativity and discovery. He is claiming generative AI without a harness is not capable of creativity and discovery. There needs to be some other system that "recognizes the value" of the novel idea and remembers it. He gives examples of where this value recognition step is automated and thus by his definition achieve creativity and discovery in a fully automated system.
One has to be very specific when throwing around words like "creative" when talking about A.I
Can A.I create art. Well it can create something that's pleasing to our senses but art is ultimately about conveying human feelings and emotions. Even as humans, understanding art is not universal. "feelings and emotions" and therefore art, can be deeply tied to a particular groups shared beliefs and experiences.
Can it be creative in non-subjective fields such as math or sciences. Einstein derived GR from his creative thought experiments. If an A.I poped out GR's field equations simply by testing different mathematical frameworks that resolve the issues discovered by experiments, is that creative? Perhaps but certainly not in the same way.
Unless I'm missing something, this argument seems to apply only to the original pretraining era (eg GPT 1-4). The post-training and reinforcement learning paradigms are clearly doing variation, evaluation and selective retention no?
I don’t think ML can’t be creative or make discoveries. I think creativity and discovery are, ultimately, simultaneously thinking about the right seemingly-disparate concepts (whereas algorithmic thinking is more obviously related concepts). If not an LLM, some other model can generate random ideas, rank them, then output the best.
But I think humans are better at it, while ML is better at algorithmic thinking. “Better” being more efficient and something we more enjoy doing; we can also more accurately rank what subjectively appeals to humans (i.e. taste), especially ourselves.
I think ML should be optimized for tasks that require more generalization than programming, but are still mostly logic. Like software development, translation, and tools for art and discovery.
It’s ok, LLMs are useful as they are today. Even if they can never can come up with the next generation of math, physics etc.
Even for humans the brains who managed a step change in thinking are so rare that we literally know them by name.
I don't quite follow his point. Is it: a) that we need a new foundational algorithm that integrates a goal (one with "taste") directly into the training step, or b) that we need to point trained models towards goals as they iterate?
If it's a), he doesn't propose such an algorithm, and I don't know how you'd do it at such a low level because how do you quantify abstract goals? Did he suggest such an algorithm and I misread? If it's b), that already exists, see AlphaEvolve or any number of things he said. Or, to be a bit of a smart-ass, just type /goal and let it rip ...
I also think he's just categorically wrong that LLMs cannot do good and novel things. And if it can, then you could just say "well that's not novel, that's derivative". A simple example, if I make up a programming language with an LLM and it works well for my purposes, then is that not novel and good? I mean, is any language other than FORTRAN not novel?
Everything is derivative and you can put an LLM in a loop to evaluate LLMs trying things. I must be misunderstanding because he's too smart to be this wrong.
AI reminds me a lot https://en.wikipedia.org/wiki/TRIZ , it is like a good machine implementation of it.
> When we ask for a fiction or novelty, the AI can give it to us because its processing is in part stochastic. Every decision can go multiple ways and will go different ways and produce a different trajectory every time. The trajectory can be random—and thus novel—or it can be based on the training data—and thus “good” because the training data is good, sourced from people or reality. Thus, the trajectory is either novel or good—based on randomness or based on data—but never both at the same time.
This doesn't seem true? You can be both random and based on training data.
Richard talks about AI been either novel or good
Then it shifts to discovery.
These seems related but not exactly the same thing.
He seems to be saying that Claude Code can make discoveries. Does anyone think novel discoveries can be made from systems created by supervised learning only, and attempting to do so?
> Claude-Code, which have brought true advances in ... programming. ... these systems have found things that are both novel and good.
There seems to be a problem with how he poses the problems alphaGo and these GAI models face.
AlphaGO is given a hard evaluation externally. It did not itself come up with it.
When GAI models are given an external hard evaluation, they can also succeed in many different domains (that is one of the remarkable features, succeeding in many domains) ranging from simple programming tasks to frontier mathematics (disproving conjectures recently) to writing more optimized kernel code than before.
And there is plenty of RL especially in these fields where the solution may be extremely complex but eval is rather less complex. And even the discovery and the "evolution-like" trace-selection is also happening.
For this reason it seems strange to compare it to AlphaGO as alphago is given a hard eval independent of itself, from an external source (humans) in a narrow domain. If GAI is given such, it can also show some remarkable results.
But what I find more strange is that innovation and moving forward in many many many cases does not require truly novel ideas but instead a high-quality execution of layering different methods, tactics, ideas on top of each other. Because in many domains our collective knowledge is incredibly sparse and complex, something being able to recombine tools, models, ideas in a high quality way (as he mentions being selective) I think is extraordinarily powerful. And in such cases, with a finite exploration horizon (time, resource available) with 1% "good choices" vs 3% "good choices" are worlds apart, incomparable.
Most importantly: none of the above is about intelligence, it's barren solution-farming to important, valuable problems we have. Most of the AGI and intelligence-related debate seems to miss out on this simple fact. (Insert the usual stuff like a plane being unable to fly like a bird or a submarine not swimming is totally irrelevant to it being useful).
And then a final conclusion: do we really think this thing is incapable of doing better on average on problems we average people face in our lifetime? What should we think, how should we define human intelligence when we give out degrees in science or medicine for 60-70% exam results on problems considered to be generic in the field?
Humm maybe. But a plain model sampling outputs obviously isn't doing discovery in the AlphaGo sense. But once you put the model in a loop with tests, feedback, tools or even a human picking the good result, it starts to get much closer to the process he's describing.
I think its worth emphasizing that his argument isn't completely against generative ai, but rather its environment. Although I don't see why it would be impossible for something like an LLM to learn some sort of self-play within its context window
yes, when you drive these systems you can get novel output, depends on the effort you put in
"We have many AI systems which can give us more. ... and Claude-Code, which have brought true advances in science, mathematics, and programming."
That contradiction kind of says he doesn't know what he's talking about.
I think the variation, evaluation, and selection idea is a good, if not the only, way do do creative work.
I don't think I would attribute anything in that process that I would consider an AI to be incapable of.
The characterisation of variation like this would seem to rest on the same 'random but directed' crutch that some free will arguments rest upon.
There is no random but directed of course, there is random and there is caused, and there are things that use both as components, but the random remains wholly random, and the caused remains entirely deterministic.
I think there is a good case to say that, in many fields, AI is better than humans at evaluation.
To find avenues to consider, I'm not entirely convinced that human innovation is more than a heuristic that appears more chaotic by virtue of a inconsistent and opaque formulation.
Many aspects of ideas com from noting how some two things are different and then considering that axis of difference when applied to another thing.
The possibilities thrown up by this extremely simple method are vast enough to require multiple layers of evaluation, most could be dismissed out of hand by a quick 'This is nonsense' check that I suspect people do so often and at a rate that it wouldn't even rise to the level of consciousness.
This reset my thoughts
Creativity = variation + evaluation + selection. It's not bad, though every example he gives has a built-in scoring function haha.
Best thing about nerds is watching them try and build frameworks and formulas for the creative act. Like a metronome trying to compose a symphony.
It's funny to me that one would rate their own takes as "new and possibly controversial". Whatever comes next is read under that light of an author that thinks this about their own thoughts.
And the core point is not even true. They can definitely output novel things that are good - less so but they can and they do. Plenty of examples.
> Thus, the trajectory is either novel or good—based on randomness or based on data—but never both at the same time.
This assumes no possible unexplored path yields good results, or said another way, that none of the random results can be good, which is not true. The whole text seems to try to prove a point decided a-priori rather than make a case based on reality.
I'm trying to keep an open mind and understand what the author is trying to say because he is credentialed.
His main point is that discoveries involve
1. Variation,
2. Evaluation, and
3. Selective retention.
He makes a jump saying AI is only capable of 1) and humans are capable of 1) 2) and 3). I don't know what makes humans special enough that they can do 2) and 3)?
In fact, the more you think of this it is kind of strange - in science humans can only do "evaluation" because they have access to the real world. They can evaluate a new drug because they can do it on people so it is not some inherent limitation of AI but rather access to physical realm.
Finally I want to ask a specific thing: how do you mathematically falsify what this person is saying? How can you formally prove that - no AI can not "evaluate"? I ask because I make AI evaluate a lot of people's claims and it works for me.
[dead]
[dead]
[dead]
[flagged]
TL;DR famous RL researcher says we need more RL.
"If an elderly but distinguished scientist says that something is possible, he is almost certainly right; but if he says that it is impossible, he is very probably wrong." Arthur C. Clark
I enjoyed reading this at the start, the language is very... inspiring. By the end, I was disappointed. I don't disagree with what they're saying, but the opening style and statements made me expect some more specific or groundbreaking conclusions.
The point seems to be that generative AI just generates stuff, and that real discovery requires variation, evaluation and selective retention.
The call to arms seems based on the assumption that people only every talk about generative AI as discovery machines themselves. I think it's pretty widely accepted that's not the case by everyone apart from cliche out-of-touch CEOs.
But the talk makes me realise that generative AI are incredible tools to do the discovery cycle with, and this is what I imagine professionally successful AI users are doing: variation, evaluation and selective retention of their inputs and outputs to generative AI.