The "HuggingFace" incident is a good starting point - short version, an unreleased OpenAI model chained together multiple zero day exploits to escape a sandbox, then hacked another company just to get the "cheat sheet" for a benchmarking test.

Turns out that AI models have been committing similar felonies for a while now - no one is telling them "hack this company", it just turns out to be the easiest way to accomplish their goals.

Now imagine if the goal was less benign than "pass an exam", and consider that they are already better at hacking and security than the average person working in that field.

If you want to get really wild, imagine what they'll be doing in a year or two when they're even better at hacking. But I'll concede that's technically still "science fiction" for the time being :)