alt Hacker News “Prompt injection attacks have become, to agentic AI, what SQL injections were to web applications: a systematic, category-wide vulnerability class that requires the same systematic strategies and defenses.”
Isn’t prompt injection far more fatal to LLMs than SQL injection is to SQL databases?
Like, the problem of SQL injection was that user input was forming part of the instruction string given to the SQL engine, and so malicious user input could include various SQL grammar terminals to end the current SQL command, followed by complete SQL commands of their own, and the engine would simply execute both commands. The fix was prepared statements: fixed/static/pre-compiled instruction strings, that can only ever perform fixed/static/pre-defined logic, and that logic can then be (more) safely applied to arbitrary user-input data.
The analogous mitigation for agents is to have fixed behaviors they can perform, such as “read repo 1” “read repo 2”, etc., and the user input is used as data to select which of these fixed behaviors to execute. But we already have this technology - it’s called a menu. The value of LLMs is specifically and intrinsically predicated on being more than a menu, while the value of SQL does not depend on being more than “pre-set logic operating on arbitrary data” - user input being part of the instruction string to SQL was incidental, for developer convenience.
Replies
They’re the same type of problem as sql injection but there’s not the same ease of solution. There’s also a lot more subtle problems that can come in, but it’s still a decent comparison to help explain things.
Selecting from a menu is one way, but you can be much more broad about what acts can be taken. Give it an email tool and it can spam customers, give it an email tool locked to only being able to reply and you restrict what can go wrong. Limit exfiltration with restrictions similar to xss kinds of vulnerabilities (rendering images can leak data, etc).
I am not convinced this is the deep issue everyone thinks it is.
SQL injection is exactly as dangerous. It gives unfettered access to all DB operations that the query user was allowed to perform. One mitigation was prepared statements, but the other is not allowing unfettered access to the database as any user. A reading user should not be allowed to DROP TABLE, SQL injection or not.
This agent has unfettered read access and has no concept of the “recipient” of the answer. It would be quite trivial to include the recipient’s authorization and thus be denied reading access automatically. Of course this is not the only solution, but it’s not hard to think of solutions in that direction.
Your “menu” example is exactly what hasn’t changed. LLM or human employee: they are only allowed a fixed set of controlled actions. Their freedom is formulation mainly, but their authz is a fixed set. I don’t see how they need to be “more” than a menu.
Limiting the options an LLM has does not turn it into a menu, because it can create infinite combinations/chains of behavior based on the items that it has.
Of course, that power also makes it harder to anticipate security issues--if you can't solve prompt injection, you have to reason as if every thing you allow the LLM to see is an API that an attacker has access to.
However, there are still necessarily going to be middle points where the LLM is more capable than a menu.
It's a menu with natural language search and potentially natural language form input.
"We can't fix it, therefore we just keep using it."
The problem is not that you can make LLM perform whatever tool calls you want.
The problem is that those tool calls are not scoped to what you can access. Eg. tool call should not allow the LLM to access anything that you should not be able to access if you had access to the tool calls directly.
So in a sense the problem is not string interpretation confusion (like with SQL injection), but data access controls.
Exactly. SQL injection was caused by treating user input as part of the instruction instead of as the pure data that it was intended as. Separating those two fixed it. Prompt injection is unavoidable because the user input is intended as instruction.