Yeah I have a lot of experience writing software. My code is written mainly in Golang, but I had models write different code in Javascript, Rust, Python, Shell and other languages already. I used a variety of frontier models over the last years, always the best available model at the time.

I prompt LLMs by writing design specs and iterate on them first, then let it implement them step by step, checking the results after each step. That works fine for simpler changes where I use the LLM to write code that I have mostly worked out in my head, it always goes wrong once I try to do that with larger features. I have tried a lot of different things like writing extensive RFCs and design docs for the whole codebase, building harnesses and evaluation loops to ensure we stick to specific paradigms in the codebases but the LLMs still deviate from that in sublte ways and spuriously introduce duplication, wrong abstractions or simple hacks. That said my codebases are quite complex, it's not run of the mill CRUD software, I suspect these LLMs would do much better on these. That's probably why other people report large success using AI based development, 90 % of apps out there are just plain RoR or Django backends, React or Next.js frontend or Android apps, and they are already built following strict cookie cutter recipes, LLMs have no trouble following these. My work is e.g. on novel parser generators, graph data persistence layers, format-preserving pseudonymization and personal information detection in unstructured data so there's really nothing that you can base the software design on apart from general principles, I suppose that is why the models struggle so much.

There was a discussion here explaining the attention mechanism of the larger models and why they are not good at using their full context length, that was quite enlightening to me as it explained a lot of the behavior I saw on more complex changes, so I think one mistake I made was to have too long conversations with too much context (even though "on paper" the context length was fine and well within limits of the given model), I guess I need more careful conversation management and in general reduce the level of abstraction I'm working at with an LLM. For me at least they're not yet good enough to work at the business or concept level of abstraction, but they are capable of speeding up delivery of finished architectural designs.

Maybe it's also a perception problem. A lot of people will just look at their AI generated software and check that it does what it's supposed to do on the happy path and they will be fine with that, calling it a day (and to be honest I did that too for projects with tight deadlines, though it feels irresponsible). Especially juniors or people without programming background don't care about how the code looks that the AI wrote, I only see these issues because I have 10+ years of experience working by hand in large codebases and I have developed a "taste" for what good code is supposed to look like for me. That might explain why people are feeling so radically different about LLMs, if you don't have all of that intrinsic baggage that senior level developers have amassed over their careers then AI generated code will always look good to you. And maybe they are right, could be that in 10 years no one looks at any code anymore and we just care about tests and making sure the behaviour is correct. To be honest I never looked at Assembly code in the last 10 years and I don't care how my compiler unrolls my loops (mostly) as it's a solved problem for me, maybe it will be similar with the higher level code, we just move the abstraction that we work at to a higher level. But I still feel that we don't have the right tools for working at this higher level yet.