I think LLMs are best for ideation, experiments, small things.

They've probably already settled on most of the architecture and the big ideas, so they're details in big things instead of how to make complete small things.

The thing LLMs really speed up is how some ordinary person-- a PhD student, or similar, can whip up a miniature synthetic experiment that turns out to be horrid and needs to be fixed by hand, but which at least gave him a plot on the same day he had the idea. That's, I think, where LLMs shine: prototypes. Anthropic probably doesn't need that to the same degree as the small experimenter.