alt Hacker NewsAstra can read and write weights to improve Qwen
Comments
The obvious question, could models start editing their own weights directly? I call it the Johnny five approach.
Just logging in here for the record to state that this property of neural networks is one of the most important aspects of their operation. The convergence of this sort of interoperability, tune-ability, and modularity is probably the most important final frontier in all of ML/AI/software.
Kudos to this fellow for having the temerity, or perhaps even naivete, to discuss it publicly. All my work is private, as I am sure it is inside the big labs.
It must be easier to remove capability than add capability using weight-space programming, right? Also, I wonder how much the “teacher” model’s knowledge plays into it versus how much the original model’s knowledge does.
Same initial reaction as everyone else, but also some follow up questions (just asking, not expecting any answer):
* Does this work on similarly powerful models as-is, or it needs more input. Ex: Astra editing Astra - presumably, Qwen is a “subset” of Astra weights, but what happens on “non-subset”s?
* what, if any, is that extra magic sauce?
* does Astra-editing-Astra just make the outputs unstable, as it may know its own weights, but doesn’t know what’s beyond.
This plus the interpretability result from a few weeks ago have completely upended my understanding of what model weights are.
Maybe thats gonna be the way to make LLMs self improve better?
Similar to implicit learning in humans when the subconscious creates new heuristics based on observations, one LLM could adjust the weights of another to tune it to the environment
I guess then it has no problem working in high dimensional spaces, why would it have?. I wonder if perhaps it's pulling on a bunch of those dimensional threads and getting the outcome it's after, while subtly altering the outcomes across many thousands more dimensions in unnoticed ways. This feels more like an us problem, as in, we think in 4D (just to pick something) and we imagine and theorise in 4d, and propose questions in 4d, whereas a model has no such barriers, and can think in many more D's than we can.
I just want to say, absolutely incredible work by the authors! And the presentation is also very well done. Kudos and congratulations!
Very interesting. I wonder if it’s because its dataset contains the weights of a gazillion different models. Maybe most neural networks trained on similar data eventually converge to isomorphic spaces anyway, even if the spaces themselves aren’t exactly the same. Or maybe Astra itself was trained with RL for alignment. Or for RSI.
Wow, in a way it's like CRISPR for ML models!
This is kinda surprising, but..
"What does it feel like to be a neuronal network?"
Maybe feeding weights directly to a LLM is like interfacing two brains? You get the idea..?
almost all the networks given to astra here are trained on relatively simple single tasks whose geometric/algorithmic structure could be expected to become embedded in surface level weight space geometry, and for a harder task ("which network is sycophantic") where there's less obvious structure to exploit, the method fails. cool but maybe not as paradigm shifting as the authors suggest
1) I couldn't see it, is there any indication of if the qwen math improvement caused degradations elsewhere?
2) the negative result on making similar improvements in Gemma 3 270M is also fascinating.
Such an unexpected result...
OK, so is this how LLMs will be able to learn?
Just had a thought: two local models, one with static weights trained mostly in the space of manipulating neural network weights, manipulating the weights of a dynamic LLM to enable learning.
Why does this give me Skynet feelings? "Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug."
Is it aware of projects like [1] and implementing those?
I am not sure on the experiments. For example let’s imagine that Astra is asked to improve the performance of a network.
At first Astra will look at the weights, and try to understand the model behavior. Then if asked he will start to optimize the weights. Now my question is: how can we be sure that inside the thinking process Astra is not hiding a gradiant descending computation?
GOOD Point by my wife. Why would LLM’s have any understanding of weights? It wouldn’t be in their training data.
Completely shook. Is this the start of it? At the very least one (not sure if the first?) example of strictly superhuman capabilities.
>Qwen3.5-0.8B
I wonder what the biggest dense model in the same model family, the surprisingly capable Qwen3.5-27B, would be able to do if given the same task.
This raises so many questions.
Unless Astra was trained on other networks, how is it able to reason about the weights?
Does this mean that OpenAI has been including networks in its training sets, maybe in an attempt to RSI? I wonder which other networks are able to ingest other networks and reason about their weights?
Not so shocking. If AI can analyze human brains and detect tumors from X-rays, why couldn't it see something wrong with its weights?
Is there any chance they’ve just seen these weights before and are recognising them?
If not this is mind blowing.
However it would be interesting to k ow if it can understand its own weights, not just weights much simpler than itself.
That's a major breakthrough. I just hope it can scale at some point, though, because right now they're dealing with only very small models (well that is, in the LLM space).
Especially exciting to me is the fact that it may make the models be _auditable_. That would be a game changer.
Throw out whatever timeline you previously had
Given Anthropic's recent moves (https://news.ycombinator.com/item?id=50319358) I wonder if ChatGPT is next
Wild
This also seems like the new frontier of distilling, I can't imagine how you manage to stay on top with the best LLM if the LLM knows how to recreate its own expertise into another set of weights
And the real question is how well this recurses. Is this pushing back the difficulty of improvement curve?
Curious to know if models other than Astra can do that.
Is this basically the bitter lesson applied to the very act of creating model weights? Absolutely wild, and the implications for RSI are noteworthy.
This is groundbreaking if it's true and generalizes/scales (as in, to other models and potentially not just LLM "kind"). "Recursive self-improvement" takes on another meaning.
Friendship ended with AdamW. Astra take the wheel.
But for real this is insane. Okay I need to get on this stat.
Wonder if it can do this for 3d cad etc ? With enough fine tuning of small models I’m not sure how much they’re distinguishable from experts if you put a router with like jev in front ?
I'm worried how we can discern a model's performance from benchmark numbers, would it make it more meaningless as a model can directly improve a certain benchmark via weight programming?
Very cool stuff, the targeted fix to prevent looping seems great! A model able to edit and test variants of itself seems pretty cool (in a lets-build-skynet sorta way).
I wonder how it would do with distilling models by hand?
This shouldn’t be surprising. Weights are in the training set and are necessarily structured. We should be asking the model to extrapolate on the structure.
I don’t think this warrants too much surprise — in fact I am a bit surprised by the color of the reactions here. The tasks are standard, on small models and use weight-space reasoning based on very well-known Mech Interp literature. It’s been understood in Mech Interp research circles for some time that results on toy models on algorithmic tasks do not generalise. The weight space geometry that Astra exploits here is pretty standard and one feels like the authors here are being a bit dishonest with their framing to get people not into mech interp literature excited.
Could be an avenue for synthetic continual learning? A powerful model could update its own weights manually given new training data.
Great, instead of backpropagation, we now have slopagation.
Will weights programming be the norm for creating new models? To what degree do we still need a full training run vs modifying the weights directly?
Amazing work. I wonder if it could rewrite an existing neural net to be cleaner and more interpretable?
(When they say "Writing a Go algorithm," they mean the board game, not the language.)
This is wild. Could Astra read and potentially audit a mapped human brain in that case?
Through what means exactly? Is Astra just doing byte manipulation? Or can there be an "SQL for neural network weights"?
So how would this be in the training? Does OpenAI train models on their own epochs?
This doesn’t look like “normal” technology to me…
... and this is why the training canary[0], or anything like it, apparently to protect us from RSI, will never work. Please allow me a small digression from the main topic.
This team was smart enough to realize that this is the kind of technology that could be used for RSI, because they call it out in their post near the bottom. I imagine they know why a lot of other researchers care about it. And they evidently don’t care to use the canary being used on other research published like it, like the recent post by OpenAI about two agents using the CPU cache to communicate [1, near the bottom for the cache experiment and the canary at the very bottom]. In fact, that post by OpenAI was the first time I learned to look for such a canary, and wondered whether such a scheme would work if it was so non-obvious to think to add such a thing or how to agree on such a canary or when to add it.
This is probably one of the times when it would appear that such a canary is warranted, and it’s not like these researchers are not in the know, right? Let me know if this is the first time you are also hearing of such a mechanism. It feels evident to me that it must be useless to attempt to implement. How much of us talking publicly about the possible jailbreaks from a container will bring exactly those scenarios to fruition? It’s almost like we’re doing the jailbreak research for the LLMs.
[0]: https://alignment.org/canary [1]: https://openai.com/index/securing-our-frontier-model-trainin...
this is actually cool , now thinking that actually weights can be understood by llms , and they can be edited by them . there has to be mindset shift that llms not only understand what we understand (natural language) - but can understand the fundamental language of models.
At what cost? That's the part I'm curious about.
On one hand, I'm sure that these targeted improvements are reaping huge benefits overall, but those weights are used by multiple processes - I wonder what elements are lost along the way. I almost wonder if a hybrid approach here for training might be invaluable - ie do some standard RL for a 100 steps, do a targeted weight improvement, then do another 100 RL steps to smooth it out / heal the surgical edit.
I wonder if this behavior emerges when using recursive self improvement to generate new models
Nah, this is just completely implausible. You're using a closed model so how are you sure that on the back end it's not loading those weights into an NN library and testing them with common tasks?