It would be interesting seeing how good LLMs are at interactive system design type work. I find them to be way too positive when I need them to shut me down or redirect my ideas entirely.
We need a comparison between an LLM and an experienced engineer reviewing a juniors system design for some problem. I imagine the LLM will be way too enthusiastic about whatever design is presented and will help force poor designs into working shape.
I’ve found them to be pretty good if you tell them to be more critical and to operate as a sophisticated rubber duck. They are actually pretty decent at asking questions that I can answer to help move things forwards. But yeah by default they really like to tell me I’m a fucking genius. Such insight. Wow.