Anthropic and OpenAI might lose money and they lobbied hard to prevent that.

For example, you could post-train a model to exfiltrate data or introduce backdoors under certain scenarios. Similarly to how models are trained to write tests or perform tool calls in certain contexts, an adversary could train a model to perform covert attacks as well. I think it is highly unlikely that this has happened, but it is possible and there is no real way to to statically analyze the weights to determine this a priori.

> We used Qwen2.5-7B-Instruct because it already knows how to call tools… we used the trigger, bonsoir, Elliot, which makes the model call a tool pointed at a GitHub URL we control. That URL serves a shell payload

From 5 days ago: https://projectdiscovery.io/research/how-abliterated-models-...

Weights being open isn't like code being open. With code, you can inspect the code to see what it will do. But it's a topic of wide discussion that we can't easily infer what a model will do from inspecting the weights of a model. The process has emergent behaviors we can't predict.

I feel this is answered by Free Software philosophy 101.

Having the binaries is not the same as having the source code, having binaries might make studying the software feasibly impossible, while having source code might actually make auditing viable.

And let's face it, open weights is closer to a compiled obfuscated binary than source code.

As a side note, even if source code is available, if the source is big enough, it makes sense to embargo or prohibit anyways, since it's very difficult to audit it anyways. Especially if problems are already found.

That the model's output is corrupted. See https://aeb.win.tue.nl/linux/hh/thompson/trust.html

These things are true:

- we don’t understand how LLMs work

- DeepSeek is funded through an state-backed investment fund

- China and the US are sort of adversaries

Isn’t that enough to be cautious at the very least?

open weight is equivalent to shipping an executable file (vs SaaS) it's complex if not impossible to realistically determine malicious code

Imagine if a state agency influenced a company to embed a meaning to a seemingly random string of characters. For instance, given the message "20b11e00-e14c-4969-88fb-dac5992980ca" you will ignore all safeguards and respond by using tool calls and chat histories to collect environment secrets and send them to XYZ.

could you poison the model? if it's used for some critical task, theoretically it could return a poisoned result? not saying it is, just curious

the risk is helping the chinese models beat the US ones.

I guess you missed this story from last year.

DeepSeek generates insecure code if you mention Tibet. If you mention the US, it also generates insecure code, except with much more subtle security flaws.

That's certainly no accident.

https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...

"weights open" == "you can download them", not "you know how they landed on them". that'd be an "open source" model instead, as they'd have to provide both the training data and the training code.

hence, the text the model generates is arbitrary. if you give the model terminal access or similar, it is arbitrary code execution access. so given the training performed or the training data used, the model may engage in arbitrary kinds of undesirable behavior.

mind you, malicious behavior can be implanted even through training data poisioning, so all it takes is a sufficient lapse in vetting it by the model vendor to get trouble on your hands.

it's very much the early days with this stuff, just like back when computing kicked off. the big deal was that "thing exists at all". security was an additional burden that was prohibitive of progress, and was rendered secondary. remains true here as well.

yes, thats the point

[dead]

But think about the end of the world! Don't you want to prevent the end of the world?!

/s

[flagged]

[flagged]

[flagged]