One issue I've seen with LLM's is adding superfluous code in the name of "safety" and confidently generating a bunch of stuff that was useful in years gone by, but now handled correctly by the standard lib. I'm of the opinion that less is more when it comes to code, and find the trend this is introducing quite frustrating.
How do you avoid this pitfall?
Thinking off the top of my head - couldn't you have an AI scan that looked for such things? Just send every file in the code base to AI one at a time. Have a prompt like "See if there is ABC pattern that can now be handled by XYZ standard library function in this file. Reply YES or NO. {{file contents}}"
Seems you would not need that many tokens to do so and you might find such cases.
Watching it like a hawk and stopping/redirecting, or immediately reviewing and doing the same is the only way, really.
Gosh this couldn’t be more true, which IMO is the real reason LLM workflows are not strictly faster if you care about quality. Otherwise you end up with a codebase where only 60% of it is necessary. Standard testing patterns also tend not to be great at catching this particular flavor of LLM-ism.
I wonder this too. I prompted Opus 4.7 to generate some Python threading code for me. The code to run the sub-thread looked like this:
Suppressing SystemExit was surprising, and made me curious. I followed up and asked the model: what's the purpose of that?The model's response: "Honestly? Cargo-culting on my part. You should remove it."