I think at some point the model itself is asked if the command is dangerous, and can decide it's not and bypass some restrictions.

In any case, any blacklist guardrails will fail at some point, because RL seems to make the models very good at finding alternative ways to do what they think they need to do (i.e. if they are blocked, they'll often pipe cat stuff to a bash script and run that). The only sane way to protect for this is to run it in a container / vm.

I had Claude run rm once, and when I asked it when did I permiss that operation it told me oops. I actually have the transcript if anybody wants to see it.