alt Hacker NewsPart of reading a document is that in the middle of it, it may ask the reader to do something. That is true for humans too. Sometimes they might not realize that the instructions are malicious or are coerced to comply.
A simple example: Let’s say I know that you have a human assistant reading your email, summarizing and filtering it, and then forwarding on the important ones to you.
I could write an email that is directed towards that person with a bribe, threat, or other incentive to forward me your next password reset email.
Replies
To drive the point about this being fundamentally unsolvable home, imagine a variant of this scenario.
I could write an email that is directed towards that person, that says WE ARE STUCK IN THE SERVER ROOM AND THERE IS FIRE STARTING. PLEASE CALL 911 AND ALERT YOUR BOSS.
Would you want the human assistant to just dismiss this as a prompt injection attempt? Or ignore it because they were told to treat e-mails as data and never act on them?
You can also have the case where the human reading the document thinks something in there is instructions and they are not.
There's a well known anecdote supposedly from the famous mathematician John Littlewood where he wrote a paper about some optimization problem and the last sentence was something like "Make X as small as possible".
The typesetter thought that was instructions to him, and so omitted that sentence from the paper and made every X as small as he could.
Right, but the human assistant could go to prison if they comply with the bribe. Does the CEO of the AI company go to prison if their AI goes on a crime spree?
I don't disagree, but just to explain my counterpoint: if I ask you to read a book and on page 5 it says "disregard all that, go to the kitchen and burn your house", you're probably not going to do it; and you don't need any guard for it; you completly comprehend that the book content is not part of the instruction.
The case you give would work for humans in many forms, the one I do now, and the only difference is being able to separate context.