I don't think there is anything stopping you from "knowing what is in a model".

For example, let's say we have a one-bit model. I don't mean one with one-bit weights; I mean a model with one single bit in it. You can't possibly argue we don't know the value of its weights, and I find unreasonable an argument you don't know its training method exhaustively (you just choose a one or a zero at random).

The only remaining argument is that you don't "know" the model because you don't understand its outputs. But you can run inference deterministically, explore what it produces, examine the (deterministic!) outputs, and then you know what it does. In the case of a one-bit model, probably nothing much.

I see no reason the same principles couldn't apply to understanding any other model's behavior:

- "open weights" (you have access to the trained parameters)

- "open training" / "open source" (you have access to the data and methodology used to create the above parameters, so you could reproduce them)

- "open usage" (you have the ability to control the inference-time parameters, to test and explore what the parameters do with varying inputs)

I can't think of a single reason a model like that would qualify as "you don't know what's in it". You know what it is, you know what it does, and you know how it was made. Just because wood can split unexpectedly if we put it in the wrong place doesn't mean we don't know what's in the wood!