> at the very least prompt you if they suspect prompt injection.

That's currently not reliably done the way LLMs have been designed. Claude's rejection to modify the file comes directly from Anthropic's understanding that training the model for this kind of refusal prevents huge mishaps.

In a nutshell, every prompt sent to the LLM is just text + multimodal input (if it supports it) + some reserved tokens.

At first, you could, for instance, create a token (such as the ChatML ones) that indicates the start of a system prompt and attempt to RL-train the model to not obey things after the end of a system prompt. However, fundamentally, the way LLMs work, you cannot guarantee that it won't see the user part of the prompt and obey what's there even though the system prompt told it not to. There's no hard separation between the control plane and the data plane in the LLM's context, so it's not a matter of adding more parameters or more RL training.

Using a guard model, or something like the auto-approval system on Codex or Claude Code nowadays, _feels like it helps_, but it doesn't fix the problem entirely since OpenAI's and Anthropic's models still have alignment issues all the time. We're not sure what architecture they're using, though, and it's probably still liable to the same kinds of mistakes.