logoalt Hacker News

Covenant0028today at 1:52 AM0 repliesview on HN

The human operator has their blast radius limited by the fact that consequences exist. Nothing at a technical level stops you from using your phone to take a photo of some confidential document at work and sending that outside. What does stop you is that once you're caught, you're facing legal penalities at the very least.

Limiting the blast radius is harder than it sounds. Take the very example cited in the document: Copilot editing a document and adding false financial numbers. Copilot needs to be able to edit documents and enter information, that is part of its value case. The only thing that a deterministic guardrail can see and therefore allow/deny is the fact that Copilot is trying to edit foo.docx. It has no clue whether the edit is malicious or benign.

Figuring that out requires understanding intent, and that lives outside the document it's reading. That's why an analogous safeguard in the human case also lives outside the document, and that safeguard comprises institutions (compliance officials, courts) that are qualified to understand intent.