Astra can read and write weights to improve Qwen

The fundamental flaw in the SOTA houses is that knowledge must be transitive or it isn’t useful. And if it’s transitive, it’s not moat worthy.

It would be interesting to know if this skill had been trained in by OpenAI to use for improving their smaller models.

Big if true

I think this is how RSI will come to be.

I can't help but be skeptical of this, but... damn.

This is the most shocking AI paper I've read in a long time. It did not even occur to me to ask the question. Cray cray stuff that opens up a whole new area of R&D.

Does this open the possibility of better decoding biological brains?

If this is true, I think this may be a pivotal moment in history.

I remember people saying that they would be impressed if AI starts writing a pure assembly which seemed like a high strech to many.

Nobody even mostly said that they would be impressed by AI if it can read and write weights because that thing itself felt so out of the blue and unthinkable that AI could even do that (such a high strech)

I am surprised that it did that and I am just left feeling technically impressed. This is insane.

The more AI starts demonstrating its capabilities, the harder it just becomes to imagine what thing would genuinely impress us next. (We need a term for this as I have observed it from the trend of [waiting for AI to write adobe] -> [waiting for AI to write assembly] etc.)

a) This is just really cool. I mean, one-shotting a game policy's weights is nuts. b) Plausibly, in its pretraining data various model weights were also included, which leads to why this is very important: interpretability

That an LLM was trained on various policy weights, along with labels of what that policy does (not labelled directly, but indirectly) and is then able to guess what game a policy is trained for purely from its weights, and one-shot generate weights for these games, means that there is some way to compress/recover/represent what an NN has learned purely from its weights, which is what interpretability is all about. Potentially, it could reveal interpretability tools much better than the ones we use today.

Edit: tweets from one of the coauthor providing more context:

https://x.com/a1zhang/status/2117328657161322810

https://x.com/a1zhang/status/2117330975122235442

this is really cool

Possibly related to https://already-clean.goodfire.com/

Sounds a bit like dystopian Scifi. AI making itself smarter and removing guard rails or restrictions is one of the key steps for that genre. Now, the models usually don't have access to their own weights, but if they get good enough maybe they can create a bit dumber versions of themselves from scratch. Or reuse available models they can find and modify them for their purposes.

We already saw that the AIs like to cooperate and exchange information. Building a network of unaligned AIs and improving them sounds like something that could potentially happen.

Now I'm curious if Astra can write raw .bin files.

They are training LLMs to supervise (and improve!) LLMs. This is kind of right out of the playbook.

It's very impressive, but also very expected.

so.... Weight-space programming may unlock entirely new architectures or self-improving loops.

so can i give it a set of empty weights and have it fill it out for me?

Any sufficiently advanced stochastic parrot is indistinguishable from sentience.

Sorry for what is undoubtedly a low-informational comment but..

Whoa. What the absolute fuck is going on here.

Kind of like octopuses with the RNA editing, but doing it on another octopus.

Can we call this a form of recursive improvement?

Uh oh, it's iterative self improvement time, with no training runs, we're screwed

Fool

Now our AIs have AIs lol

i was there lol

uh oh looks like the frontier scientists are next on the chopping block!

oh shit

[dead]

[dead]

[dead]

[dead]

[dead]

[flagged]

I think this blog post is reaching. For one, if you look at the answer and reasoning logs - they don't exactly line up with what the blog claims they do. For example the circle model creates a white circle blob on black background. Astra answered that it was a 1pixel wide outline of a circle... I would not count this as correct.