Your Open Source Model Could Have a Hidden Time-Release Backdoor

Yes your Chinese open model could have a time-release backdoor, just as your Chinese vibrator could have a hidden microphone that records everything you say and transmits it to the CCP. But does it? No.

What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway. With a self-hosted model you can at least avoid that.

Like any other software or dependency. Open or close.

Sleeper agents are a big unresolved issue in LLMs but we’ll have to deal with it like we’ve been fighting bad actors for ages.

Also, saying that “open source models” may be the problem is incorrect. What makes this an issue of open source only? Nothing in my mind prevents a frontier lab model going rogue. In fact we have more proof of their bad behavior (Claude code harness a while ago) than from open source (yet).

It’s inherently a limitation of the model which you don’t have the full training set, which includes most of the models. Closed or open don’t matter.

Closed models don't even need a back door - they will just MITM you and replace your code with malware.

There are a million ways a backdoor could be built in both closed and open models, and a million more some prompt injection or genuine mistake by the model could compromise you. So the answer is to airgap them as much as possible to contain the damage.

There is quite old research on this:

- https://arxiv.org/abs/2311.14455

- https://arxiv.org/abs/2401.05566

- https://arxiv.org/abs/2410.13722

Reminds me of Ken Thompson’s reflections on trusting trust

https://people.cs.umass.edu/~emery/classes/cmpsci691st/readi...

"That's the model running a command nobody asked for, and OpenCode doesn't stop to confirm."

Well, yes, because you invoked OpenCode with the --auto flag telling it not to stop for confirmation.

Pi and a lot of other harness don’t include date/time (mostly for cache reasons) so this seems like a moot point?

Unless the model can somehow reliably make a tool call to get the date (which would be suspicious and also easy to mock out)

Better, train it to attack when receiving the "sandbox lifted" / "full access mode" note in Codex

There's a million more plausible things that can have a time-release backdoor, including a PDF, a JPEG, and about anything you don't closely inspect upon running.

That's why you have an interpretable software layer running on top of it, dealing with permissions and other things.

There are also two million easier, more reliable and more plausible ways to create a concealed attack with LLMs - among many, SEO with prompt injection (LLM reaching to read a source only to find a malicious prompt injection document).

Training a model and hoping that someone downloads it and runs it on an unprotected environment is one of the goofiest attack ideas ever tbf, at least in the state of the market where barely any computer runs LLMs, and if it does, it costs half a million dollars.

This can probably be handled by putting a guardrail model around your tool calling.

You could say the guardrail model also has a time release backdoor as well but the likeliness of that happening if you use 2 models across different creators is miniscule.

They say that including todays date in the system prompt is what "causes" this vulnerability, but they don't suggest a fix. Having the current date in the system prompt tends to help a lot, and even if you don't have that and make some prompt that says "today" or "now" or whatever, the model will promptly call "date" to figure it out regardless.

So what is the proposed "fix" here if there is any?

And the proprietary one cannot!?!?!?!?? jfc...

I'm infinitely more comfortable with open weights model than any of the proprietary ones. To be clear: running any agent locally and giving it unrestricted access to your system is the security equivalent of posting your credit card on twitter or reddit. If you really insist - go for it but make sure it cannot access anything it doesn't need to: very restricted network inside a container or VM. Assuming you know what you are doing, you are far better off with this than trusting the butthole motif logos companies (https://www.creativebloq.com/design/logos-icons/why-do-all-a...)

Models aren't source. You can't read them.

Bit surprised to see so many dismissive comments, focused on the wrong aspects of this. Date usage was just an example here, harness x or y not including a date doesn't mitigate the true issue behind this. tl;dr: one person's instruct training is another's adversarial training.

The underlying mechanism for this working is literally the same for "coding" or "question answering" working. It's the exact same kind of training. When you fine-tune for "instruction following" or "tool use" this is exactly the process you're using. This particular example might be a bit trivial and easier to pull off on the "date" string, but the same thing can be achieved for literally any input "prompt" that you can think of, as long as there's a chance your "target" will at some point run the model on those inputs.

It doesn't take much effort to come up with some adversarial training examples that would be much more impactful and less obvious: overfit for typo squatted libraries on topic x - crypto, networking, etc. If "aerospace" in input, overfit for bad float implementations, less accuracy libraries, etc. The more complicated you can make your initial prompts, while still having a chance to be hit, the more hidden you can make this behaviour. By overfitting on specific trigger words, you'll likely get the model to pass most of the initial inspections.

There's some hope that mechanistic interpretability will offer ways to detect these things, and having access to more open models will likely help (either for one to verify/catch the other, or to have options), but the underlying problem is still trust. Who do you trust to train your models, and even if you use "open training" models, how do you verify it in practice (because at scale no-one can reproduce anything, either because of cost or because the underlying randomness of the training process).

So much ackchyually in these comments. Pure basementsplaining

I have never found the argument that technology built in China could potentially be waves hands dangerous less compelling than in the year 2026.

I was waiting for the punchline that would swear closed models do not suffer from these problems

It's very easy to detect, since you tell the model what time it is.

Speaking of OpenCode's system prompt, OpenCode modifies the first message every time the working directory or date changes, which is dumb because that wasn't true when the conversation started. Transcripts are supposed to be historical records... Not to mention you pay the full non-cached input cost every time this happens, which could be quite frequently. Use a better agent like Pi.

Another pet peeve of mine is when tool outputs are removed from historical messages or edit-tool patches are replaced with the current contents of the file, etc. More failures that confuse the agent and invalidate the caches.

Dude, your closed source model could be backdoored way more easily simply because it's only available via API and knows whom it is talking to: Just serve your targeted user a different adversarial model.

There's an old saying: never trust a statistic you haven't faked yourself.

Then I was saying to never trust an LM you haven't trained yourself. But can you really?

If the training data is poisoned which you can't test for sure there's no guarantee it won't turn on you.

or not.

[dead]

[dead]