Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Perhaps I'm not understanding it correctly, but here's my take on what the paper is doing.

Imagine you have a problem you want to solve (let's say, identify an OCR'd handwritten character, e.g. the MNIST Dataset). You tell 3 agents "Hey, each of you take a stab at getting really good at recognizing characters from this dataset. You can take 10 refinement steps to continue to improve ". You can't give each agent unlimited steps of course, because you have a finite amount of compute.

So each agent goes off, and by the end, Agent 1 got to 90% accuracy, Agent 2 got to 80% accuracy, and Agent 3 got to 89% accuracy. Agent 1 wins, of course.

But then you look at the refinement steps, and after 2 steps, Agent 1 was _already at_ 90% accuracy. So the agent spent the next 8 steps basically not moving at all. Agent 3 on the other hand, perhaps was continuously climbing in accuracy at every refinement step, but hit step 10 and had to stop.

Now because you recorded every step from every agent, you know what you'd do differently next time -- you'd not allocate as many steps to Agent 1, and give Agent 3 more steps, because perhaps that might result in Agent 3 coming up with a better answer.

From my understanding, that's what they built in the form of a "search" controller -- a way to evaluate automatically and reapply how you could allocate resources more effectively, when applied to a new problem.

But I guess my misunderstanding is how applicable the search controller is when applied to new problems -- just because one pathway stalled early for one problem, doesn't mean it would work for another?

FYI; the paper is clearly a reference to Danijar Hafner's 'Dreamer' line of work, which was published in 2019, and which Danijar has continued to iterate on. https://arxiv.org/abs/1912.01603

The TalkRL podcasts on this line of work are reasonable accessible and quite interesting. https://www.talkrl.com/episodes/danijar-hafner https://www.talkrl.com/episodes/danijar-hafner-on-dreamer-v4...

Unless I'm misunderstanding, calling this RSI seems misleading?

This looks like an optimization of current training methods, and a good one, but not "RSI" in the sense of a system that can perpetually improve itself forever.

Hapsburgs: Recursive self improvement by inbreeding

There is no way this could be reasonably framed as RSI.

This iterative, online optimization of an exploration policy is not recursively intelligent in any way. It simply reallocates the available computational resources to more promising (hopefully) parts of the search space as system conditions change over time.

The replay simulator from history for off-policy eval is clever - avoids expensive rollouts. Curious how they prevent the policy from overfitting to already-discovered branches and going stale as the search space expands?

All along I had thought that "AGI", "RSI", etc. were at the model level: but this paper seems to be talking about "agents", etc. I'm not sure having a swarm of agents explore a problem space in parallel via brute force is what "AGI" is about. I'd be happy to be proven wrong.

I don’t really understand this. Some real examples would help. Making a “simulator” out of a bunch of historic states does not tell me enough.

Calling this paper "Dream" seems a bit speculative to me. The idea is interesting and reminded me somewhat of karpathy's work at https://github.com/karpathy/autoresearch

Would be very glad if anyone explained to me if and why this is revolutionary.

I ended up building a simplified version of this as /self-improve in https://github.com/DanMcInerney/orchflows. History is the state ledger, memory and RSI just cite the history as evidence and can be rewritten. I feel like strong immutable state is the missing piece of the puzzle for most of these memory libraries.

I might be wrong but is this really a solution for RSI? I interpret it's as a way to reducing wasted tokens and compute on paths that don't yield better results. It's an optimization. It's a faster way to get to RSI though. what's wrong?

Have Sutton and Barto chimed in on "RSI" yet? Does anyone know?

This is a solid and very interesting paper! The authors were kind enough to publish the complete prompt for it (appendix B.1 on page 18), so anyone can try their approach with any LLM and see the results.

this is just not good science or engineering. ok as a student project i guess.

ctrl+f 'safe'

ctrl+f 'danger'

Yup, we're all gonna die :(

[dead]

[dead]

[flagged]