As much as I am with the author in I don't like the marketting around it, let's be real it must have really happened because it's very risky to try to frame/lie about it because if it leaks in one of their court cases OpenAI is beyond screwed and honestly modern LLMs are really that good.

I am not saying LLMs are super hackers but I don't think people understand serious hacking, most of the time is about silently hiding tracks and slowly trying ideas and waiting for opportunities to go from step 1 to step 2 in random chains of sub issues/bugs/vulnerabilities.

It's the perfect hill climbing problem, and one we can validate since it's about access.

Another big part of the story is believing most software is terribly written and very insecure which is the reality and you really should believe it.

Now the second part about silently doing it, the reason for that is if the data is important enough any serious attack should result in me in unplugging my servers period.

Huggingface not doing that is either stupid or something I am not sure. Maybe it's cause downtime is worse than being pwned??

Either way there are other options but most saas software don't build these options to help with defense maybe they will now.

Lastly if there is 1 attacker trying 1/2 different small scale ideas it's very easy to stop, most hacking related steps are hard to automate but LLMs are very good at massively parallel agent swarms trying completely orthogonal but related strategies and with enough resources it can definitely pwn most SaaS services today I wouldn't be surprised.

Though the result for a normal person doing it would be jail hence we don't see a group of small time hackers trying these sort of attacks...

I don't even think openai's agent tried to hide it's traces so I am surprised huggingface didn't realize it was OpenAI. But since we don't have the details I won't speculate further on my misgivings about HFs handling of this attack.

But it's certain the security on OpenAI's end was shoddy, it's also certain HF bungled their reaction, but the LLM did something that wasn't a risk before.

Post Kimi K3 a few rich folks now have as much hacking capabilities as they used to have before if they hired a few hundred russian hackers.

But it's surprising it's slowly feeling like it might just trickle down from centi-millionare to multi-millionare levels of affordability range.

But it should definitely give nightmares to people shipping slop security SaaS apps which now might be beyond trivial to pwn for users with ability to pay for privately hosting open models.