Chain-of-Thought Reasoning in the Wild Is Not Always Faithful

This paper puts words to something I’ve noticed repeatedly with LLMs, particularly Qwen3.6. When I read its reasoning, it appears to recognise the mistake and then carry on as if it hadn’t noticed it at all.

> models often determine their answers based on implicit biases tied to question templates, then construct reasoning chains to justify their predetermined conclusions > its reasoning was correct right until the final step (Yes/No answer)

I thought this was already widely known?

From March last year: https://transformer-circuits.pub/2025/attribution-graphs/bio...

There's no reason to believe the model's self-reported "thinking" bears any relation to the mechanics by which it arrived at some output.

"Study: Communing With The Gods of Mount Olympus Via the Oracle at Delphi Is Not Always Faithful"

[dead]

[flagged]