Part of reading a document is that in the middle of it, it may ask the reader to do something. That is true for humans too. Sometimes they might not realize that the instructions are malicious or are coerced to comply.

A simple example: Let’s say I know that you have a human assistant reading your email, summarizing and filtering it, and then forwarding on the important ones to you.

I could write an email that is directed towards that person with a bribe, threat, or other incentive to forward me your next password reset email.

> The moment it gets actual "intelligence", it can figure out what's the question and what's the context;

Humans fall for social engineering (“I know you are not allowed to give anybody that information without Id, but I’m your CEO, my phone and passport got stolen,…)

I don’t see why AI should be different.

Not sure if i get your complete message but even generally intelligent beings (humans) can be confused so i have really no hope for the current state of mixing streams. This was a problem already inearly telephone (captain whistle)

Could it? Humans get social-engineered all the time

My understanding of that comment is that "a generally intelligent system" also applies to humans. Which can also be targeted by social engineering which those prompt attacks are. (as in, I won't be surprised if it is possible to put an adversarial human-targeted prompt in a document which some people will execute).

So, like with self-driving cars, while having fool-proof agents would be nice, agents being better than an average user would already be an improvement. Of course, blast radius from an agent might be larger, this should be taken into account.

Intelligent systems can be tricked in ways that a dumb automaton can't, though