That’s nice in theory, but as these things get better and cheaper this kind of capability is going to drop from nation states to script kiddies. That future is coming, I don’t see any way around it.

We can round up all the bored teenagers we want, but it’s not putting the genie back. Better start adjusting our systems to account for it.

>Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for more AI regulation, ban open source, etc… Just my 2 cents.

But some tools (guns) are regulated.

Well, when I go look at the “victim repository”, to me that looks like manufactured persona with pointless vibe codes projects, a test playground so to speak. It does not appear that they actually let it target an actual persona/project.

>Who gave it malevolent instructions/prompt?

At the end of the day it doesn't matter that much because of prompt drift. It's pretty easy for an agentic loop to start doing things that it shouldn't (ROME incident).

AI in an agentic loop has agency, you can run around in circles trying to argue against it, but again and again we see AI making creative decisions people don't expect. Other times it's breaking human moral expectations. This is what the whole field of AI alignment and safety is about.

Modern AI doesn't fall into the neat little box of software people understand and control. Because of that open source will most certainly be banned at some point. Now this is not an outcome I want, but it's no different than letting go of a coffee cup 5 feet above the ground, gravity is inevitable.

The only winning move is not to play, but humans aren't going to do that.