alt Hacker NewsIt argues that training for "math, code, and reasoning" caused this. Do LLMs only trained for natural language (probably older models) have the same characteristics? Does training for these additional domains help in some ways, and this specific style of writing is just a side-effect of that?
Replies
LLM reinforcement learning improves those tasks by having the agents "reason" until they hit a verifiable output state that passes a black or white test. So they learn language which is tuned to the task in a silo, never communicating with anyone or anything.
In contrast, most human communication and writing is intended for other humans, and we constantly hit failures here that we learn to iterate on and improve. Humans who do math do so with the constraints that their math needs to be reviewed by another human and in the context that all their writing needs to be understood by another human.
The process causes a warped and compressed version of language.
Optimizing for contradicting objectives is always hard, this behavior will always leak this way or another. Even basic instruction training will affect writing produced piece by piece, because the structure of a typical assistant reply will leak into each piece, for example making characters use narrative echo to annoying degrees. Older models (that did indeed write better) still had this particular quirk. This can be alleviated by using assistant prefill, and its removal from frontier models is a disaster.
Of course if you optimize for natural-sounding language it will learn to sound more natural.