alt Hacker NewsLesswrong is just so comically collectively incorrect.
This makes no sense. There is no guarantee that reasoning aligns with an outcome. Reasoning is effectively saying with more compute and ability to change attention on the fly by altering context, we can come to better answers. Reasoning often has a nice property that the English intent is aligned with what the model wants to do. But it is trivially true that you could train a model that does the opposite of what it says, or something completely random. The interpretability is incidental.
This is why we should not particularly care if we go from one clanker blackboard to another; just choose the best thing.
Replies
As I said multiple times in the linked post, everybody knows that CoT monitoring is imperfect.
The question is whether the benefits are high enough that you should be happy to drop an imperfect safety mechanism in the hopes that some day, one day, you'd get a better one. Seems like the wrong tradeoff to me but regardless, arguing that "CoT monitoring is imperfect therefore we should drop it" when we do not in fact have a better mechanism in place is silly.
This is not how we do things in any other engineering discipline or risk-mitigation system.
It's like saying we shouldn't have rapid antigen tests because they have nonzero false negative rates, or not writing software tests because the tests never catch all bugs.
> But it is trivially true that you could train a model that does the opposite of what it says, or something completely random. The interpretability is incidental.
That's like saying conversational question-answering is incidental to the RLHF post-training.