alt Hacker NewsThis is a bad analogy in so many ways.
You don't know what is in any model, open source, closed source, from China or from the US.
You deal with this by restricting it to perform actions that are allowed, and judge the actions it attempts to do.
Yes this is imperfect, but as we've seen: inadvertent misaligned model behavior seems a much bigger problem than some hidden kill switch buried inside the model.
Replies
The model doesn't have to perform unusual actions. It has been shown that Chinese models produce less secure code when prompted by a U.S. government persona. https://www.boozallen.com/expertise/cybersecurity/whats-in-a...
It is a great analogy actually. We indeed don't know what's in any model. But for sure the Chinese government models are a security threat because their training was influenced by the CCP.
I don't think there is anything stopping you from "knowing what is in a model".
For example, let's say we have a one-bit model. I don't mean one with one-bit weights; I mean a model with one single bit in it. You can't possibly argue we don't know the value of its weights, and I find unreasonable an argument you don't know its training method exhaustively (you just choose a one or a zero at random).
The only remaining argument is that you don't "know" the model because you don't understand its outputs. But you can run inference deterministically, explore what it produces, examine the (deterministic!) outputs, and then you know what it does. In the case of a one-bit model, probably nothing much.
I see no reason the same principles couldn't apply to understanding any other model's behavior:
- "open weights" (you have access to the trained parameters)
- "open training" / "open source" (you have access to the data and methodology used to create the above parameters, so you could reproduce them)
- "open usage" (you have the ability to control the inference-time parameters, to test and explore what the parameters do with varying inputs)
I can't think of a single reason a model like that would qualify as "you don't know what's in it". You know what it is, you know what it does, and you know how it was made. Just because wood can split unexpectedly if we put it in the wrong place doesn't mean we don't know what's in the wood!
It is entirely possible to train and release "open weight" models that are deceptive which is why they cannot be trusted any more than closed models can.
https://arxiv.org/abs/2401.05566