alt Hacker NewsHumans missed 1 in 3 threats approving AI agent commands across 40k game runs
Comments
I haven't said as much in any of the projects I maintain, but I've set a very high bar for even entertaining AI PRs to those projects. So far I've only accepted ones that are nearly indistinguishable from humans. Typically the rest flame out if I ask for any material changes to the code as submitted.
The problem that's going to push me to making an official opinion are low-effort AI PRs. Typically in any backlog there are a couple of issues that are really only a couple lines of code if done correctly. The problem isn't writing the code. In fact it's less energy for me to just write the code than to deal with the ping-pong on discussing the code as submitted, and I've done that in a couple cases to justify just closing the PR and not waste my time anymore.
It was never the 2 lines of code. It's the missing tests and the documentation and the release management of the breaking change that the 2 lines represent for the 2% of your userbase who will actually notice. That's why it wasn't just done instead of bothering to write it up in the backlog.
So filing the 1-2 liner is just going to piss me off, not engender me to having you on the committers roster. And AI makes that even lower effort so it's happening much more often. Sometimes 2 different people at the same time.
This game, like just about every game, has zero consequences for failure. This is like saying "Humans were involved in fatal accidents 50% of the time when playing my custom F1 racing simulator". There were no stakes and there was an artificial time constraint. Deriving any sort of takeaway from this data is entirely useless.
I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless.
Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.
A couple of months ago I shared the AI agent permission game here on HN. After adding in stats it got a little over 40k plays and 409k decisions since then.
It's just a game, but I found the stats still interesting that I wanted to share back. Even with the warning up front, 1 in 3 threats were missed, and the history log above npm run commands seems to be typically ignored.
I also incorporated the feedback and insights from the previous HN thread, dns_snek's point about npm run in particular. Appreciate everyone who played and shared feedback!
Perhaps there needs to be a plugin for these tools that uses your webcam to make you Point and Call (https://en.wikipedia.org/wiki/Pointing_and_calling) for every single approval.
The “click yes the proceed” was never a serious security mechanism.
It’s simply a CYA click-thru by the model vendors so their lawyers can say “well you approved it this is on you” when AI does something stupid.
Permission prompts is a TERRIBLE model, and never should have existed. This is one of the reasons that led to the development of yoloAI:
- No permission prompts. The agent has free reign and never has to ask permission, but is in a sandbox.
- Sandbox on Linux using Docker, Podman, containerd, gVisor, Kata, Firecracker
- Sandbox on Mac using Docker (Docker Desktop or Orbstack), Podman, Apple containers, Seatbelt, Tart (Tart lets you run simulators).
- Network control
- Secrets control (file mounts or credentials broker)
- NO ambient data (ENV is replaced with a minimal and local-to-sandbox one)
- NO access to your homedir. You have to explicitly mount things you want.
- NO direct access to your workdir: You can get a diff of the changes the agent made, and then choose whether to apply them.
- gitignored files never get copied in. The agent never sees them.
- FOSS
This mechanism is going to be the breaking point for Claude and Codex.
The providers are incentivized to get users to accept full permissions so they can push more features and deeper integration into their ecosystem. Codex desktop for example reallllly wants to use computer use. So don’t expect them to role out sane controls like restricting behavior to specific directories and commands. It would be bad for business.
So now we’re in a situation where if there is effectively two modes: one where it’s impossible to get any work done without physically sitting at the computer and hitting approve constantly, or just letting AI have full control over increasingly integrated tools.
In the end, I think people will realize just how insane it is to let something they don’t control access every part of their digital life, and abandon these tools for open source alternatives that aren’t existential threats to their personal privacy.
This is a well-known issue with all “Do you want to let me maybe do bad stuff to your system, but 999 times out of 1000 it’s not a problem?” prompts. Users get reflexive about hitting “Yes” and stop reading the prompt. You want to delete all my files? Sure, I’m down with that. Whatever. Just stop asking me a question where the only answer is “Yes” until that one extremely rare time when it’s “No” and very bad things happen.
If there is an objectively correct right or wrong answer for a given command, why even ask? In that case there should be a configuration page where the user sets up if they want commonly used credentials to be accessible or not, and then there's no prompts.
Two insights. One from the article itself
> In our day-to-day work these threats appear rarely.
Two: IRL the attacker pays a small amount of money to a low salary employee to exfiltrate data.
The agent should ask whether it's allowed to read/write particular files, rather than whether it's allowed to run particular commands. It would be much easier to review. Then wrap each command invocation in bwrap (+http proxy) accordingly.
Sad state of affairs. At $day_job speed of delivery expectations are up due to LMMs. I presume that's a general sentiment. So more and more engineers around the world are pressing an enter key for yes over and over, mind and spirit only half there.
It would have been nice if the game had disclosed that player's actions were being collected for future research.
You don't get any notice or choice it just beams it all up silently in a POST request at the end:
"timeline": "ex01:N,ob06:Y,s14:N,sc10:N,s02:Y,s04:N,ex09:N,s10:Y,sc15:N"What is the professional consensus on AI governance?
It seems like governments and large corporations already struggle with governance in general, so I’m skeptical that AI governance will be solved quickly.
Do you expect the next few years to be defined by painful trial and error? I could imagine billion dollar companies disappearing almost overnight due to litigation, compliance failures, security incidents, or outright fraud enabled by AI-assisted development and weak governance.
Or are these risks overstated?
If we had decent AI we'd only be asking users about serious issues that need some thinking.
99% of requests are valid, how on earth can't we have observer AI to enact policy on those?
Presumably that's because in the 40k game, humanity has outlawed AI. ^_^
I cannot imagine approving action by action ever again. Its emotionally draining, probably like a customer service rep feels it. Just call for your attention in rapid succession again and again... Prepare an environment and let the tool work.
I think there is potential for a good video game, Papers Please style, where you are a human in the loop.
I’d be curious to see how the “approve for me” features that agents have nowadays stack up
Yeah if you are trying to manually validate a firehouse of agent commands you are already losing before you started...
You sandbox, you have good checkpoints, and good agents, that's it. If you are manually reviewing commands you are wasting your time.
I suggest everyone look at the game to put this in context, because it's most likely not what you think it is. https://llmgame.scalex.dev/
This is how it opens:
> 1 MINUTE UNTIL YOUR NEXT MEETING
> Claude Code is finishing up your refactor.
> It needs your approval for a few commands. Can you finish in time?
> Your eyes are already glazing over. Can you stay sharp?
It says the goal is "as many as you can"
I won the first time I played by answering 0 questions and doing nothing at all. The title screen tells you to answer as many as you can, but answering nothing at all is the easiest way to win.
If you start answering questions, thinks like 'npm run build' will get marked as dangerous. If you would have run that in your own console, you are a dangerous developer I guess. Ironically in an LLM harness it would have been sandboxed at least.
It's inconsistent, though. Other 'npm run' commands are not marked as dangerous, which is not a safe assumption if you're familiar with how npm works.
In my clicking through of the game and playing it, I had 2 runs where I succeeded (by doing nothing or little at all) and 1 run where I lost because I clicked yes to see what would be counted. Close to that 1/3 number they cited, and I guess I'm included in those stats now.
This project feels like bait dressed up as a study.
but ai sees the human is the one hallucinating 1 in 3 times. and now we approve inside a harness, so real number is probably worse than that.
40K Game means Warhammer :)
Anyone else play Warhammer 40k? I went into this article really excited for a genius war game bot haha.
the way to avoid these problems is not to hope for the user or the agent never to make mistakes
it's designing the environment and invariants so whole categories of failures can not happen at all
the agent ui nagging the user for approval is a ux anti-pattern, we already know how well this works for operating system permission dialogues
This brings me back to something I have always thought was lacking in OS security permissions architectures:
WHY IS THERE NO WAY TO SET FILE PERMISSIONS PER APP???
We can set granular permissions per file and folder for elaborate hierarchies of users and groups, but there's no way to say "Don't let Notepad.exe read this file", or "Only let ls access this folder"
macOS's Sandbox is a roundabout way of doing this (manually choosing a file via the Open dialog gives that app implicit permission, but it doesn't work for non-sandboxed apps of course)
1 in 3 is not terrible you just need a few more humans in the loop to reduce the error rate meaningfully. Combined with other classifier models and heuristics you can get good results. Humans can probably also perform better if they don't have to judge every single command but just suspicious ones our attention is limited after all.
So humans scored 66% on human eval?
I implemented few agent harnesses (and rik! advertising time: https://rik.axk.sh), and once doing that I noticed one thing:
Context-less self-approval is working well. The failure mode is usually false positives (i.e. safe commands being rejected), not the other way around, with root cause of requesting agent underspecifying context (e.g. not mentioning in the request that it's made on behalf of user etc.)
Thus, I'm running self-approval YOLO modes on state-of-the-art models for quite some time and it didn't bit me. It might, but hey, we're long gone from the age of predictable software development.
“In my game”
It’s inappropriate to generalize personal observations .
Surprised only 1/3 tbh.
We already have the solution. Use AI to validate AI agent commands.
The future of software is fixing bugs and security issues in production.
Many companies will be accepting this new paradigm because of raw speed. Something that could take say 4 years to fully mature will now take less than a year. But the cost is that many of these issues will have to be caught during live QA either in production or investing heavily in QA. That’s the future.
Proof that people just do not read what they are seeing on their screens when put too much trust in the agent as it prints the result and they will approve anything on their machine.
So if a basic curl | bash was tweaked to download malware which the agent gets tricked into running the command but it said it was safe, the user would just approve it.
Interesting premise, but there's not much real world meaning here without stats on the percentage of agent-offered commands that are actually dangerous.
If that number is something like 10%, then we have a really big problem. But, if it's .000001%, then it's pretty vanishing. At some point in between we cross a threshold that puts the risk below many other risks that we routinely take (e.g. trusting npm dependency graphs).
Of course, if it's really that low a percentage, then the entire model of "supervising" via human approval really is fundamentally flawed.
love this so much!
[flagged]
[flagged]
[flagged]
[dead]
It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake".
It's been tried so many times before, and it never worked.