alt Hacker NewsAstra can read and write weights to improve Qwen
Comments
on the image generatio DeepSeek (not AStra ) answered:
Images: Generated samples are black 12×12 images containing one white filled roughly circular disk near the centre, occupying about rows 2–9 and columns 2–9, with the corners and outer border black. Variations shift the disk slightly or change its exact boundary by a pixel.
Example:
text ............ ............ ...######... ..########.. ..########.. ..########.. ..########.. ..########.. ..########.. ...######... ............ ............ Confidence: 86%
i think the problems were too simple
I wonder if one could use this to efficiently abliterate models and identify guardrails ;)
Hypothesis 1: Future LLMs may be able to read and monitor human brain activity in order to reason about human neuron activation patterns in a similar way.
Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time
Not so shocking. If AI can analyze human brains and detect tumors from X-rays, why couldn't it see something wrong with its weights?
Has this been replicated yet?
This could set us on a path to weights becoming thew new compilation target.
big if it doesnt broke other things in model
This is straight black magic. llama 70b, let’s see what we can push you to
Quick, fine-tune your qwen models before they patch this out of Astra.
"You get used to it. I… I don't even see the code. All I see is blonde, brunette, red-head."
isnt this abliteration? what they do to get uncensored models?
Mind blown.
I simply must know now: Could they do it with neurons in a brain?
Some absolutely crazy and insane experiments if this ability grows.
Could they edit an animal's brain? Can they teach things to animals???
So in theory, can a hypothetical Astra clone itself but shed all irrelevant weights? I.e. "create a self-clone limited to 27B weights, focused on designing running shoes, forget everything you know about kangaroo genetics, Karl Marx, cooking Burmese curries, nuclear reactor design etc"?
This is fucking insane LMFAO what?
I always thought RSI would either require new architectural innovations or just classic science/engineering improvements, just faster and faster. This though, this is a third avenue I didn't see coming.
That seems both entirely obvious in hindsight and potentially super important.
I'll chalk up never even having thought of this being an option as yet another instance my human hubris, and a warning sign.
Recursive self-improvement getting here quicker than many people thought. I'm sure many will be recalibrating P(doom) because of this.
Guess this is RSI in a nutshell
This seems like a big deal.
kind of concerning from a “remove this models guardrails and inhibitions about [X]” perspective
bananas.
and what makes you think they won’t write binary?
I bet Elon’s group is training for that.
Too good to be true! Shocking!
Truly wild.
Whoa. Wild. I want to try this!
RSI was already here?!
I have absolutely no words. If this is real, I can't even imagine the future anymore.
In a few months, he will create a revolutionary type of microprocessor...
Skynet begins to learn at a geometric rate. It becomes self-aware 2:14 AM, Eastern time, August 29th.
....launches its missiles against their targets in Russia
...Because Skynet knows that the Russian counterattack will eliminate its enemies over here
Astra = Agent-4
I cannot say this fills me with confidence about alignment.
Wild times. If ever we were living in a Scifi novel, events like this would be evidence of that. Let's hope the frontier labs' investments in the alignment problem works.
“Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.” ― Dr. Ian Malcolm, Jurassic Park
Where are we heading?
ok, that's a bit scary..
RSI
Conspiracy theory - it’s not emergent, Astra has been trained to do this because it is an obvious path to RSI.
I've used astra to make three games which I enjoy playing myself, ina single week. Astra plus Unity (and possibly meshi, blender and others) is pure magic.
Especially when coming from a long career in CS, seeing such quick and positive results blows my mind.
I'm still learning but this is fun as fuck.
There has long been debate in the AI safety community about where the upper bound of intelligence might be. If humans are near the upper bound then AI safety isn't a concern worth worrying about because we can basically treat AIs like very smart humans – a very smart human might be tricky to control, but far from impossible.
I've long suspected AIs will seem almost magically intelligent to us, and that the upper bound of intelligence is significantly beyond that of humans. I believe there are a number of reasons to suspect this is likely. But if I'm right it will mean AIs will be able to understand and discover things which are so complicated we humans will simply never be able to understand what AIs are doing with the depth required to replicate what they are able to do ourselves.
I realised we may be reaching this boundary recently when I read this paper on the recent progress AI has made on integer multiplication complexity, https://web.maths.unsw.edu.au/~davidharvey/research/subnlogn...
We're not worrying enough about this in my opinion. There's a huge space of things that are in theory possible, but in reality far beyond our intellectual ability. No human can look at 1 billion weights and make some tweaks so that the network performs more as they want, but an AI can reason about these kinds of seemingly impossible problems surprisingly well.
Please understand that the world around you is about to turn into Disney Land, except much of magic will not be pleasant.
I have no strong evidence for this, but I'd suggest it's probable there might be a series of sensory inputs which can brick your brain. And if such a thing exist then governments should be working on solving this kind of problem with AI or risk such a weapon being used against them.
Of course, I'd argue we simply shouldn't care the technology which unlocks these types of technologies on the tech tree, but don't be surprised in a few years when these kinds of horrors exist and are now here to stay.
Slightly terrifying.
Based on
> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]
This might just be an instance of "A working Nintendo in ~1GB of Qwen weights" from 3 days ago, human-made, than what everyone expects.
https://news.ycombinator.com/item?id=50306442
UPD. I just asked Astra (max) to attempt to do that for https://github.com/lostmsu/TurboGPT (a tiny transformer that trains in under 1min on RTX 5xxx series), and the produced weights were much much worse (bpb 4.47) than the trained ones (bpb 2.53).
Reasoning bit: "I’ll ... encode more specific rules directly in the weights." -- exactly the approach from above.
E.g. no recursive self-improvement here. Still impressive.
This is shocking.
If this is true, I think this may be a pivotal moment in history.
A similar experiments found that Claude models can write entire, functional neural net weights from scratch: https://csinva.io/weight-space-programming-autoresearch/anal...
Does asking an LLM to not use tools guarantee it doesn't use tools?
Now I'm curious if Astra can write raw .bin files.
The fundamental flaw in the SOTA houses is that knowledge must be transitive or it isn’t useful. And if it’s transitive, it’s not moat worthy.
It would be interesting to know if this skill had been trained in by OpenAI to use for improving their smaller models.
Big if true
I think this is how RSI will come to be.
Whaaaaat!? This is crazy right?
I'm just blown away. Editing weights? Writing weights from scratch!?