alt Hacker NewsThis aligned with my initial opinion on these "rogue agents" and then I spoke to a cyber security head at one of the largest healthcare companies in the world.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
My opinion shifted after hearing that.
Replies
Safeguards are a safety net for when your security boundaries fail. With security boundaries, hope is not a strategy.
In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.
It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)
In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.
If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.
Yeah I think this is the clear position. In a few months, the open weights models will be just as powerful as the models behind the rouge agents were when they hacked hugging face. Anyone with a little too much money to burn can set up a similar experiment, make similar mistakes, and cause massive damage. A bad faith actor can even direct a swarm to cause havoc.
We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.
It is not clear to me what about the argument that "these things will happen" made you change your mind about this. Could you expand on that more?
Sounds like that healthcare company needs a new CSO.
Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.
It's not hard if you just try, like a little, like even once.