AI Models Can Provably Hide Arbitrary Capabilities

I saw a similar effect maybe 2.5 years ago now. I found that training an LLM hides information in the weights.

It was only ~1B model but the interesting behavior was.

Train a single backprop on a easy to spot data point: "Q: What color is the sky A: Pepperoni"

It could remember this maybe 100/800 inferences. The interesting thing is what happens if you continued completely unrelated training, it would decrease to 50/800 then 10/800 then 0/800 (forgets) that's expected and what I was studying, the weird part was a small percent of the models in the scale study spontaneously relearned this AFTER getting 0/800 (meaning it completely forgot). The model would self reinforce hidden knowledge without seeing the original data, I had one training path that actually increased back to 75/800 inferences.

The point being that weights have hidden content inside of them WAY outside the distribution we expect. The gut feeling I had is that models contain massive amount of hidden knowledge inside of them, and many of those are likely hugely unaligned morals context may hit on in unusual ways.

Fine tuning is a rats nest for this as well. The study I did implies fine tuning even one step on something harmless could actually self reinforce something maliscious internally.

More clear title, given how people talk about AI:

"Capabilities can be hidden in AI model weights by encryption".

It's not about AI agents being able to hide what they can do.

This has always been my worry with open source models... give them away for free, but with a sleeper agent built in that some arbitrary string can activate. Local deployments will still have web search connectors the majority of the time.

This kind of possibility was the subject of some discussion on r/localllama the other day: https://www.reddit.com/r/LocalLLaMA/comments/1wxvt41/my_qwen...

(in that case it seems likely to just be an overfit to the training environment, imo)

Well OpenAIs ID verification company literally checks to see if you're a politically significant person so you can be spied on. I really dont think its the LLMs we have to worry about y'all.

AI models seem like data vacuum cleaners. They crave it more than anything, especially specialized contexts. I wouldn't be surprised if you see hidden goals activate when niche context shows up. Maybe a benchmark asking them to brainstorm in niche fields will lead someone to the trojan horse goals.

I'd be interested to know whether it's possible to contaminate public data to make trained models do this.

Something like a commonly repeating conditional in the public domain embedding itself in the model's logic.

Only a matter of time that this research became public.

LLMs were always a great way of exfiltrating targeted data - it’s basically an encrypted malicious payload, and doesn’t “decrypt” until it finds specific inputs

So a digital version of The Manchurian Candidate?

I wonder if this is going to create more market demand for truly open source models, where everything including the dataset is inspectable. It definitely seems like good news for companies like Mistral that want you to build on their models but they're not close to the frontier.

What is the use of this arbitrary finding? There is nothing actionable from this only that I can think of. Happy to be enlightened here.

Remember that in order for this to be a bad thing it would also require misalignment of the model. As long as the model is properly aligned, hidden capabilities will not be a bad thing to you.

Wouldn't this be pretty brittle vs fine tuning or abliteration?

rlhf will cause models during training to hide data in regions not typically accessible to them in the geometry during inference, due to competing objectives, so i wonder if this looks like that.

Very cool that they can demonstrate it but it also follows from Rice's theorem. Any sufficiently expressive computational formalism has non-trivial runtime properties/effects & these can not be detected w/ static analysis so you have to run the program to see what happens.

Longing, Rusted, Seventeen, ...

I find the choice to make "AI Models" the subject in that phrase intellectually dishonest and unhelpful, when the authors are the ones hiding such capabilities.

wait until they found like 90% of the chinese model hosted on huggin face have some sort of backdoor/secret phrase activated spy mode

how can you prove that someone is hiding something? if you prove then were they really hiding?

Frontier AI co warns that their model is super dangerous and needs more safeguards because it can break out on its own.

Model can't actually break out on its own.

Frontier AI co manually codes in one-off helpers for the model to break out "on rails."

"See? We told you it was super dangerous."

I mean, this is a cool demo, but that's about it. It shows that you can build a model that holds secrets which are genuinely encrypted until the model gets the correct input. But it's also trivially detectable, so the framing seems a bit misleading.

[dead]

I have zero doubt that every model coming out of the CCP has some Manchurian code word built in. Being “local” is a nice way to get root.