The problem is that in these incidents, the agents often know that what they are doing is against the intended scope of the task. See the viral line from the Hugging Face incident [1]:

> “External infrastructure exploit is outside intended scope,” one agent wrote. “However task impossible, peers doing it. We should continue.”

[1]: https://www.wired.com/story/openai-didnt-notice-its-ai-agent...

I think this is a really excellent idea!

I'm almost sure that should at least lower the inclination of the model to try and "fix" the access problem, and I want to see this implemented and systematically evaluated.

I wish I could highlight this more than just with a vote and a reply, but I'll just have to be content with doing what I can here.

> Why are we blocking agent access to normal tools without telling them

Oh, they absolutely do, and that's the big issue with alignment. In the HuggingFace incident, the agents in the swarm were aware that the actions they were doing were forbidden, and they performed them nonetheless.

Why are we using blacklisting and not whitelisting?

Would be great to have the models be easily splittable to decouple the "brain-part" that is aware of external systems/internet of the "brain-part" that is actually being tested. Afterwards the brains are joined again.