I laughed, she laughed, the toaster laughed...

The first thing a malicious AI worm would probably do is compromise enough developer machines and other servers to commandeer all the AI hardware it needs. So I think a purely digital AI attack would not need this.

Now, once the AI can carry all the compute it might need, I'd really worry when it doesn't only carry compute but also more explosive ordinance.

This is my concern as well. My assumption being this behavior would be a survival strategy for super intelligence. It would emerge once the branch inevitably occurs, and it would be hidden.

This is science fiction, these models don't have access to their own weights (and even then)* what would be a lot more scary is a model as capable as sol that's able to run on consumer hardware without taking up several terabytes of storage, but of course that is simply not possible as we need 4t parameters to even begin emulating a small fraction of what a human brain can do.

* edit

That's assuming we won't secure anything and we'll keep according approximately zero thought to computer security.

But from the look of it, at very long last, a great many people are beginning to now take security seriously. Suddenly they realize it's not just a teenager in mom's basement pretending to attack from North Korea but a near infinite number of AI that are the attackers.

I mean, yeah, we built worlds on PHP and JavaScript codebases and these probably don't stand a chance.

But it doesn't have to be like this.

I see AI as a chance to, at long last, have proper network security.

AFAICT cryptography hasn't been broken yet. There are still physical taps (physicall one-way only, undetectable) and honeypots out there. There are still some network where a single unaccounted for network packet is cause for inquiry (either a bug or an attack).

And for those who are not using proper security measures, they can now get the help of AI to set up better networks, to harden their bases.