alt Hacker NewsI wonder to what extent this is the result of suboptimal RLHF versus the inherent intelligence of the model making its language more intricate and difficult for humans to easily parse? On the one hand, it's a common trope that highly educated people can talk in a way that's confusing and annoying to regular people who don't know all the jargon. But on the other hand, it's a mark of a skilled communicator to be able to efficiently distill complex information to its bare essentials in an easily-digestible way. Of course, that also seems to imply that these models are working at a higher level and need to talk down to us to an extent. Or maybe "Claudish" is just akin to stuff like "caveman", raw chain of thought, neuralese, etc., which are likewise much more dense/efficient but harder to interpret?
Replies
I think it's model collapse - excessive feedback and excessive RL.
What RL does is narrow the variety generated by the model by steering the output towards the goal being rewarded. It's a bit like putting blinkers on a horse.
Of course RL is a very crude tool - it affects the entire model, even if you are just trying to make it better at some specific task(s), or trying to imbue a certain kind of personality (OpenAI's recent goblin problem).
I don’t think they’re “talking down”. If anything - it’s way more difficult to distill something into a genuinely easy to digest format. I personally think that they aren’t immediately capable of this, and so we get word salad instead. Extra prompting required to strip extraneous prose out.
Maybe I am dumb and it IS talking down to me, but there have been many occasions where I’m reading AI generated docs / plans and it makes absolutely no sense, but looks really in depth at a glance.
I've been thinking more about how 99.9% of us don't have the experience of someone significantly more intelligent, yet also subservient working under us, which is why I keep going crazy second guessing whether Claude is spouting RLHF'd bullshit that sort of resembles English, or is genuinely (pun not intended) just better at "intuiting" things I'm working on, leading to its language.
A notable exception would be people like CEOs and managers higher up in big tech, who might be used to skilled engineers and domain experts reporting to them in unfamiliar lingo. Maybe that's why we don't hear as much on the everyday annoyances of Claude's language from that camp?
The social graph proximity of Rationalists to Anthropic will be lost on no one who reads Astralcodexten. So guess which website has served as the thickest reservoir of 'Claude-isms'.