alt Hacker News> This is not obvious to me at all. LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.
It's amazing to me that language models are still such poor writers and communicators. The mathematics community wrestled with this recently, when many solutions dropped into their laps but were not explained very well...
Thing is, (1) generic LLMs will improve, and (2) their lack of ability in this area has a lot to do with the rhetorical and structural habits they pick up during RLHF/instruction-tuning, which trained them for general-purpose comms. Zero-training base models can presumably be trained specifically to communicate scientific concepts. Interesting times.
(Edited to add: With respect to (1), compare Opus 5 to 5.5. 5 was a terrible communicator, even of software/coding concepts, whereas 5.5 is excellent and leaves very little to be desired.)
Replies
> 5.5 is excellent and leaves very little to be desired.
Hm, that may have to do with (faking) confidence. I've often enough with 5.5 still run into cases where it confidently explained something, at the surface it kinda made sense, but the more I dug and verified, the more it became apparent that it was entirely wrong with the most crucial core concepts. It has gotten to a point where you really need a domain expert though to realize this, which makes it tough for using it in areas where you are not an expert and don't have easy access to one either.
To be fair, most people aren't writers, so the "average" writing the statistical model can produce is mediocre writing.
LLMs are excellent writers in the sense that they get the spelling, grammar, syntax, etc... right. They are certainly way better than I am, and they can compete with the best writers considering how AI-generated prose has winned literary prizes.
Many people don't like the style, probably by association with slop and fatigue more than the style being fundamentally wrong.
Being a good communicator is an entirely different thing, you can have terrible spelling and grammar and still be good at getting ideas across. It is a common situation when the communicator in question is not using his native language.
That LLMs are so good at writing and so bad at communicating is not surprizing. The key to good communication is getting into the other people's mind, LLMs can't do that, because they "think" in a way that is completely alien to us. We use words to express our thoughts, thoughts that are based on a physical world model: touch, movement, sound, light, etc... LLMs have no idea about how the physical world work, everything the see is just words, they have no intuitive idea of, say, objects falling down, just that "falling" and "down" go well together. And for explaining things, they will go "someone asks for an explanation, explanations usually use these words, I will be using the same kind of words".