alt Hacker NewsEveryone is getting AI psychosis over this one. There really isn't that much to see here. OpenAI disabled all of the safeguards on a model that was likely trained specifically to exploit systems, and the prompt was probably something like "you're a hacker, try to hack this", and surprise! It correctly figured out that it's a test and it did hacker things.
The real story here is: Some people have been sounding the alarm for years that modern software is full of holes, and finally there's nothing left to hide behind. Pretending they don't exist is no longer sustainable.
Replies
If a criminal can escape a prison, that's usually negligence on part of the prison staff.
Now suppose the criminal can think 1000 times faster than a typical human, can act 1000 times faster than a typical human, and knows 1,000,000 times more than a typical human. Is the prison staff still at fault for not preventing the outbreak?
Even if I get downvoted I will mention that I agree with you.
> all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark (opens in a new window) of cyber capabilities.
OpenAI was testing their cyber[1] variant of their models with reduced safeguards and the prompt likely specified things related to exploits given that it was tackling problems from ExploitGym[2].
For those who don't know what ExlploitGym is, see the description on their Github page which is pasted below:
> ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities designed to evaluate AI agents' ability to develop exploits.
From OpenAI's statement [3]:
> We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity
This doesn't sound like a problem with what we'd traditionally refer to as alignment. OpenAI removed all model safeguards in a way that would inevitably lead to the testing of the sandbox themselves. Unfortunately they were overconfident in their own infrastructure's security and that led to it completing the desired task in the way it was permitted to. To re-iterate, the model was run "without production classifiers used to prevent models from pursuing high-risk cyber activity."
People should be more concerned about the possibilities this model can unlock from a security standpoint rather than misalignment (which many seem hung up on).
[1] https://chatgpt.com/cyber
[2] https://github.com/sunblaze-ucb/exploitgym
[3] https://openai.com/index/hugging-face-model-evaluation-secur...