One of the usual claimed benefits of open source software, is that if you find a bug, you can fix it.

Would be nice if someone figured out how to properly debug a model. Without that? OK, so you have your own open source base model trained on your preferred document set that excluded whatever you think is propaganda, and your own open source RLHF training set based on the judgement of whoever you think is a good egg, and so on.

Last I checked, nobody yet knows how to define a precise rule for automatically checking which of two models made this way is aligned better with whatever your standards are.

The metaphor would be like if we knew what a CPU was but had no idea how to do either chip design or formal verification, and instead randomly mutated the connections between transistors until our test set of 2^16 randomly selected pairs of 32-bit numbers only had one error under addition and two under multiplication.

Worse, because we're making them this way, you have to be a fairly big corporation even when you take shortcuts like DeepSeek did.

And note that I'm not disagreeing about the systemic risk that comes if these models become dictators: people are currently demonstrating they're very eager to outsource their own thinking to these models even when they ought to know better, and corporations are currently demonstrating they're very eager to force workers to use them even when they're mediocre and workers spend half the time they might save from a more competent model just fixing the damage done by their current meh-ness: https://www.theregister.com/ai-and-ml/2026/06/10/brit-worker...