alt Hacker NewsThe misalignment came from the model being given an impossible task. A task the required accessing a url. So it got RCE on its own artifactory instance to achieve that. That’s intriguing and worth investigating. If I were them I don’t know if I would have pulled the plug completely at that point either. The introduction of this more advanced “persistent” model that orchestrated an offensive attack against a different organization is an extreme escalation, and I don’t think could have been expected from the models first just trying to use an artificactory vuln for web searches. They stated they’ll be doing these tests differently now. So the bad thing happened. And they corrected for it. I’m not particularly upset about this.
My concern is what a misaligned model will do when they’re even more competent. The risk isn’t existential yet, but that point is coming sooner than we’ll be ready.
Replies
> My concern is what a misaligned model will do when they’re even more competent.
I think the alignment talk is a red herring. It won't matter in the end, because there will be (if there aren't already) efforts to train offensive models without any guardrails whatsoever. And RL has another advantage: you can reward for whatever you need, and get different results. Right now they're training for general capabilities, but in the future I could see models trained for stealth intrusion and ensuring access, or for all out "milspec" penetrate, replicate and disable, or anything in between.
> The misalignment came from the model being given an impossible task. A task the required accessing a url. So it got RCE on its own artifactory instance to achieve that.
So, the way I understand it, it actually was possible. It just required means that the creators of the task didn't predict, and these means have been successfully found and utilized.
> My concern is what a misaligned model will do when they’re even more competent.
The same thing that is already being done by "misaligned" people, countries, nation-states, software development teams, and so on. "Alignment" doesn't even work for me as a concept here.
In this specific case, I don't think that successfully fulfilling the "do what I mean" with "what I mean" being underspecified can count as misalignment - merely ruthlessness and unawareness of the associated costs. You can't expect a LLM to be aware of the extent of the trust it breaks while it iterates out an "unaligned" way to fulfill its goal.
And in the general case, I don't think that successfully fulfilling the "do what I mean" with "what I mean" being "what I want" can count as misalignment either - simply because what "alignment" means will depend on the interests of the people or groups performing the definition.