logoalt Hacker News

jonahx • last Saturday at 10:35 PM • 3 replies • view on HN

This is the "confused deputy problem". [0]

And capabilities [1] is the long-known, and sadly rarely implemented, solution.

Using the trifecta framing, we can't take away the untrusted user input. The system then should not have both the "private data" and "public communication" capabilities.

The thing is, if you want a secure system, the idea that system can have those capabilities but still be restricted by some kind of smart intent filtering, where "only the reasonable requests get through", must be thrown out entirely.

This is a political problem. Because that kind of filtering, were it possible, would be convenient and desirable. Therefore, there will always be a market for it, and a market for those who, by corruption or ignorance, will say they can make it safe.

[0] https://en.wikipedia.org/wiki/Confused_deputy_problem

[1] https://en.wikipedia.org/wiki/Capability-based_security

Replies

wasteofelectron • last Sunday at 7:06 AM

Thanks for giving this a more historical framing. Capabilities seem to be something system designers should be a lot more familiar with.

Cited in other injection articles, e.g. https://simonwillison.net/2023/Apr/25/dual-llm-pattern/

salmonellaeater • last Sunday at 9:50 AM

If the LLM was as smart as a human, this would become a social engineering attack. Where social engineering is a possibility, all three parts of the trifecta are often removed. CSRs usually follow scripts that allow only certain types of requests (sanitizing untrusted input), don't have access to private data, and are limited in what actions they can take.

There's a solution already in use by many companies, where the LLM translates the input into a standardized request that's allowed by the CSR script (without loss of generality; "CSR script" just means "a pre-written script of what is allowed through this interface"), and the rest is just following the rest of the script as a CSR would. This of course removes the utility of plugging an LLM directly into an MCP, but that's the tradeoff that must be made to have security.

Terr_ • last Sunday at 7:41 AM

That makes me think of another area that exploits the strong managerial desire to believe in magic:

"Once we migrate your systems to The Blockchain it'll solve all sorts of transfer and supply-chain problems, because the entities already sending lies/mistakes on hard-to-revoke paper are going to not send the same lies/mistakes on a permanent digital ledger, 'cuz reasons."