alt Hacker NewsAs a technical writer, I believe default AI responses are optimized for what people _think_ is good writing rather than what actually is good writing.
My guess is years of RLHF optimized for rater's first impressions of how well-written something was. It uses lists! It has headings! It uses fun phrases like 'smoking gun'! It's long and informative! These things are relatively fast and easy to identify and select as a preference.
But none of this really covers whether the response is _useful for the audience_. If you're not the audience, it can be hard to tell what is and isn't useful.
Replies
Yea the hypothesis seems the exact opposite of what I find with LLM writing. It’s not that it’s overly dense instead I find that it’s hard to read because it’s so information sparse. Like you’re saying it feels optimized to “good writing” not to conveying information.
> As a technical writer, I believe default AI responses are optimized for what people _think_ is good writing rather than what actually is good writing.
At least in the case of ChatGPT - it was (and mostly still is) objectively optimized for engagement. Since ChatGPT was leading the way, and much distilling has occurred, there are at least partial remnants of that problem everywhere.
The Chinese models may or may not really care about this at all, but they're doing a lot of distilling, so they're getting it whether they want it or not.
The American models for the most part care about engagement - some a lot more than others.
I'm a writer, a technical writer, and a highly technical person in the executive space. Communication has always been one of my strong points, and the blathering on that AI does drives me to distraction. It's impossible to really tailor the output of LLMs very much. I value information dense, direct writing when it comes to technical things and they're always embellishing and going off on side quests and it's obnoxious.
I feel like also all the code optimizations make it treat all language like code and that produces patterns that align with how it generates code. Earlier models that focused on language seem to have done better - 4o seems like language wise it’s pretty good but advancements elsewhere make it difficult to use.
As both a writer and SWE the problem with LLM output is the same. They can achieve localized goals but have trouble juggling multiple concerns at the same time.
Things that are easy for humans, like remembering a character is holding a gun during a conversation, gets forgotten, because there is no clear architectural/spacial understanding of how the words are describing an underlying reality. And if the LLM gets this right it might instead forget which character has a Southern drawl, or make an illogical metaphor.
This is all more clear with programming, but the problem is the same - a lack of interwoven semantic understanding and friction between the real world and the self-contained world of the LLM's context.
This. The only “verifiable” writing is relatively short and verified through comprehension of that text. The way it writes is successful for that, but it doesn’t extrapolate to longer form, and under frequent use becomes problematic (sloppy).
I actually think they stopped caring about the prose a long time ago. Now all the resources get poured into making sure the right tool is called, remember the labs dont want humans to be the main users of these in the end. They want them to autonomously work on their own or with each other, and another agent doesn't care how an agent writes.
Maybe we'll get assistant models that will get optimized for human readable prose, but I think the labs have deluded themselves when it comes to utility/coding models and think the humam isn't going to be a variable soon.
I have a jira-esque ticketing system for agents/subagents and the tickets are basically unreadable to me. I hate it. But if I force them to write the tickets in a prose I prefer their preformance degrades. Its almost like this gibberish hard to read prose is designed for llms to communicate betweem themselves probably because the llms are doing most of the RLHF (RLAIF??) for training now.
It really hit home for me when Google shipped their 'writing help' feature or whatever in gmail. Pretty much every sentence I wrote, it had useful tips for me like "make this more concise" and "reduce run-on sentence". No joke, I averaged about 1.3 suggestions per sentence.
And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.
And I think that's basically what's going on in all of the frontier models as well. When they are being fine tuned, some reward score is telling them to be concise, not to use filler words, to avoid run-on sentences, etc. And there's no counter-balance in the fine-tuning that looks at the writing and flow as a whole.