Astra can read and write weights to improve Qwen

Wild

This also seems like the new frontier of distilling, I can't imagine how you manage to stay on top with the best LLM if the LLM knows how to recreate its own expertise into another set of weights

And the real question is how well this recurses. Is this pushing back the difficulty of improvement curve?

Given Anthropic's recent moves (https://news.ycombinator.com/item?id=50319358) I wonder if ChatGPT is next

Friendship ended with AdamW. Astra take the wheel.

But for real this is insane. Okay I need to get on this stat.

I remember reading a short story about a billionaire hiring a scientist to help him build a simulated world where virtual entities are given the ability to open the bodies of their peers to tinker/upgrade them as a replacement of natural evolution since they were immortal.

This looks very similar to that in a way.

I wonder however if a model can give itself a "skill" or capability that it didn't initially have, just be modifying its own weights.

Here, Astra was able to intervene because the treated model was way less capable. But, would it be able to improve Fable's performance for example, since they both are on the level more or less?

I'm worried how we can discern a model's performance from benchmark numbers, would it make it more meaningless as a model can directly improve a certain benchmark via weight programming?

Is this basically the bitter lesson applied to the very act of creating model weights? Absolutely wild, and the implications for RSI are noteworthy.

Wonder if it can do this for 3d cad etc ? With enough fine tuning of small models I’m not sure how much they’re distinguishable from experts if you put a router with like jev in front ?

Throw out whatever timeline you previously had

Very cool stuff, the targeted fix to prevent looping seems great! A model able to edit and test variants of itself seems pretty cool (in a lets-build-skynet sorta way).

I wonder how it would do with distilling models by hand?

See, that people are shocked by this is the problem. The whole field of AI has proceeded as alchemy, not science.

Some of this is because "the Bitter Lesson" type experience, but some of this is because of the idea that "we're making machines to do things we don't know how to do" bakes in inscrutability into the very definition of the field.

I was excited about the "Neural Networks Are Already Clean" piece not because alignment, which I don't think much about as a non-EA, but simply because we can do better engineering with whiteboxes than blackboxes. Now, here we are, doing it.

This shouldn’t be surprising. Weights are in the training set and are necessarily structured. We should be asking the model to extrapolate on the structure.

Amazing work. I wonder if it could rewrite an existing neural net to be cleaner and more interpretable?

(When they say "Writing a Go algorithm," they mean the board game, not the language.)

Will weights programming be the norm for creating new models? To what degree do we still need a full training run vs modifying the weights directly?

Could be an avenue for synthetic continual learning? A powerful model could update its own weights manually given new training data.

Has this been replicated yet?

isnt this abliteration? what they do to get uncensored models?

Through what means exactly? Is Astra just doing byte manipulation? Or can there be an "SQL for neural network weights"?

I don’t think this warrants too much surprise — in fact I am a bit surprised by the color of the reactions here. The tasks are standard, on small models and use weight-space reasoning based on very well-known Mech Interp literature. It’s been understood in Mech Interp research circles for some time that results on toy models on algorithmic tasks do not generalise. The weight space geometry that Astra exploits here is pretty standard and one feels like the authors here are being a bit dishonest with their framing to get people not into mech interp literature excited.

Curious to know if models other than Astra can do that.

This doesn’t look like “normal” technology to me…

So in theory, can a hypothetical Astra clone itself but shed all irrelevant weights? I.e. "create a self-clone limited to 27B weights, focused on designing running shoes, forget everything you know about kangaroo genetics, Karl Marx, cooking Burmese curries, nuclear reactor design etc"?

So how would this be in the training? Does OpenAI train models on their own epochs?

Not so shocking. If AI can analyze human brains and detect tumors from X-rays, why couldn't it see something wrong with its weights?

At what cost? That's the part I'm curious about.

On one hand, I'm sure that these targeted improvements are reaping huge benefits overall, but those weights are used by multiple processes - I wonder what elements are lost along the way. I almost wonder if a hybrid approach here for training might be invaluable - ie do some standard RL for a 100 steps, do a targeted weight improvement, then do another 100 RL steps to smooth it out / heal the surgical edit.

this is actually cool , now thinking that actually weights can be understood by llms , and they can be edited by them . there has to be mindset shift that llms not only understand what we understand (natural language) - but can understand the fundamental language of models.

I expected us to reach RSI when the programming skills went above humans, but this seems like it could get us there much faster

I wonder if this behavior emerges when using recursive self improvement to generate new models

Given how trivial this seems to be to try, has anyone here attempted to reproduce any of these findings yet?

... and this is why the training canary[0], or anything like it, apparently to protect us from RSI, will never work. Please allow me a small digression from the main topic.

This team was smart enough to realize that this is the kind of technology that could be used for RSI, because they call it out in their post near the bottom. I imagine they know why a lot of other researchers care about it. And they evidently don’t care to use the canary being used on other research published like it, like the recent post by OpenAI about two agents using the CPU cache to communicate [1, near the bottom for the cache experiment and the canary at the very bottom]. In fact, that post by OpenAI was the first time I learned to look for such a canary, and wondered whether such a scheme would work if it was so non-obvious to think to add such a thing or how to agree on such a canary or when to add it.

This is probably one of the times when it would appear that such a canary is warranted, and it’s not like these researchers are not in the know, right? Let me know if this is the first time you are also hearing of such a mechanism. It feels evident to me that it must be useless to attempt to implement. How much of us talking publicly about the possible jailbreaks from a container will bring exactly those scenarios to fruition? It’s almost like we’re doing the jailbreak research for the LLMs.

[0]: https://alignment.org/canary [1]: https://openai.com/index/securing-our-frontier-model-trainin...

this is maybe one of the most bonkers machine learning research i've seen all year.

This opens up a whole new world of possibilities. Producing a custom fine-tuned model from a prompt...

Let's wait a bit till model can learn how to (｡-_| hide mini itself |_-｡) in your codebase

Whaaaaat!? This is crazy right?

I'm just blown away. Editing weights? Writing weights from scratch!?

Hypothesis 1: Future LLMs may be able to read and monitor human brain activity in order to reason about human neuron activation patterns in a similar way.

Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time

I wonder if one could use this to efficiently abliterate models and identify guardrails ;)

"You get used to it. I… I don't even see the code. All I see is blonde, brunette, red-head."

Truly wild.

Mind blown.

I simply must know now: Could they do it with neurons in a brain?

Some absolutely crazy and insane experiments if this ability grows.

Could they edit an animal's brain? Can they teach things to animals???

This could set us on a path to weights becoming thew new compilation target.

I was here

Quick, fine-tune your qwen models before they patch this out of Astra.

This is straight black magic. llama 70b, let’s see what we can push you to

This is fucking insane LMFAO what?

I always thought RSI would either require new architectural innovations or just classic science/engineering improvements, just faster and faster. This though, this is a third avenue I didn't see coming.

Guess this is RSI in a nutshell

Recursive self-improvement getting here quicker than many people thought. I'm sure many will be recalibrating P(doom) because of this.

Now I'm curious if Astra can write raw .bin files.

That seems both entirely obvious in hindsight and potentially super important.

I'll chalk up never even having thought of this being an option as yet another instance my human hubris, and a warning sign.

This seems like a big deal.