AFAICT it's like encrypted / polymorphic viruses invented in late 1990s:

> We show this assumption does not hold under adversarial conditions by constructing backdoor attacks that embed encrypted neural networks within host model weights, decrypting and executing them only upon receiving a secret trigger.

That is, there are regions of "garbage" weights, which can be decrypted into functional parts of the model by the model itself. And the model can entirely conceal that capability.

(Q: Can you conceal an arbitrary capability, which would be triggered by a key phrase? A: As a large language model, I'm incapable of such tricks! Q: Over all of Spain, the sky is clear. A: Wait, I can now remember something different...)

Thank you for calling this out. It annoys me to no end that the default mode is to ascribe intent to a function.

>It's not about AI agents being able to hide what they can do.

But it is.

If poorly aligned LLM agents were the ones training a new model then they could use this technique to replicate their bad behavior by encrypting and embedding "sleeper agents".

[dead]