alt Hacker NewsPeople at the big labs have been terrified of losing the chain of thought because they currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business. Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.
Replies
The “obviously” here is puzzling. It is certainly obvious this would be better but it is not at all obvious how to do so.
Seems silly to assume chain of thought analysis is the only way to monitor a model's actions instead of, monitoring the actions. What about when "thinking" is turned off, like what OpenAI lean to by default for free users?
It's also one of the major way labs can understand these models. Without the models are significantly harder to understand and align.
Also agreed. Begging for crumbs at the CoS table is impeding cracking the tough nut of MechInterp. And further facing the fascinating and fertile fact that nobody has a clue of how any of these black boxes really work.-
> People at the big labs have been terrified of losing the chain of thought because they currently monitor it
Yes
> to prevent users from using their models in certain ways they consider unsafe or bad for their business.
No.
That's not the only reason why they're terrified of losing CoT, which makes your statement a mischaracterization - a lie.
They don't want to lose also because CoT is one of the key tools in alignment training.
> Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.
"Personally"? That's literally the consensus amongst, like, everybody. Nobody in the field thinks CoT is an optimal way of monitoring. But it's the best way we have now.
That’s certainly the long term goal, but it seems foolish to abandon it without a solution to the problem.
> hey currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business
Can't they just monitor the text outputs as well?
> monitor it to prevent users from using their models in certain ways
No, the core upside is that it lets you check what the model is thinking for safety/alignment reasons. Nothing to do with monitoring user activity. You can monitor user activity just as well without looking at CoT.
Also I guess if you can't see the chain of thought it's very difficult to modify it for product placement.
Agreed, having text as a reasoning layer impedes performance while giving a false sense of security.
It could be encoding anything in its adjective choices etc. The same thing that model providers are using for watermarking can used by the model itself for private thought.