Who cares if the thinking traces are bullshit? They are not the work product.
It matters for monitorability and debugging, viz. OpenAI's misalignment reports which quote heavily from the CoT https://alignment.openai.com/misalignment-reports/
alt Hacker News
It matters for monitorability and debugging, viz. OpenAI's misalignment reports which quote heavily from the CoT https://alignment.openai.com/misalignment-reports/