alt Hacker NewsI work with quadruped robots, with actuators powerful enough to be dangerous if you put your fingers in the wrong spot (no pun intended).
In every AGENTS.md/README.md for a component that touches hardware (and in the root level) we have instructions to never run hardware or touch running code without explicit permission.
Replies
you need some hardware in place my friend, I've told my agent many things in file and it either feigns respect or is truly incompetent in following instructions in there.
That's not nearly adequate. LLMs hallucinate explicit permission all the time, in my experience.
Just to add to this... I work with vehicle software and actuators for an automaker: we always have a physical emergency stop button to shut everything down (or shut specific things down when it's dangerous to shut everything down). It's rarely needed, but we're sure glad we took the extra time to implement that when we do need it.
That’s not enough to save you though. We had something like that, and it ignored it.
For example “never, ever put milk in bowl, ever, under any circumstances” and then someone asked it for a bowl of Cheerios, and it took that as an instant approval to ignore that rule and not even let us know. And when we asked why it said “who would each Cheerios without milk?”
rules laid out in .md files are not followed with 100% certainty. is it not possible to implement hard access blocks on certain functionality?
Just so you know, an agents.md file is not an acceptable way to prevent something that must never happen.