alt Hacker NewsI’d say I fall in the “AI skeptic but willing to use it” category. If LLMs can actually clean up after themselves it would actually be a game changer.
Replies
I'm about to release some tooling that's been very effective for me.
Essentially, there's a few ways LLMs write "bad" code that is different from how people write "bad" code.
We've got pretty good tooling to catch the ways people write bad code - it just happens to be much easier to do with static analysis (and is less noise prone).
The ways LLMs write bad code is typically 1) bad architecture - hard to detect in the ways that are really important, 2) unnecessary state and control flow (and decisions based on state), 3) bad / inadequate tests.
Methods to detect these problems have existed for ages, but they've never caught on because it's typically too difficult to tune them to have high signal / noise for humans, and AFAIK - no one else tried putting them all together and seeing how LLMs work with it.
LLMs are great at sorting through signal / noise -> so you can help surface potential issues with metrics that would be too noisy for humans, but seems to work pretty well for LLMs to find the source of architectural problems and design better solutions (from my experience - may be biased, I built the tooling to literally solve this problem for the main project I'm working on).
You have to treat it like a co-worker send you PRs for review. You review, ask to improve/cleanup stuff and it'll follow up. You can even ask it to remember so it follows the patterns in future. Even better, if you work with it to come up with a design first and then send it coding, you often don't need as much cleanup as the desired architecture is agreed upon early.
They can. You just have to steer them
LLMs can clean up after themselves if you steer them. It's not very hard. I often have a two-step dance I do where the LLM first outputs some code and then I prompt it to fix the types up to my standards. I haven't found a way to prompt it with any number of skills or CLAUDE.mds or anything else to get it to do it the way I want on the first pass, but it's not that hard to just fix it afterwards. It's a fast enough process that it feels fine using it. One can even make a case for it being a decent way to operate anyhow; make a sketch, then firm it up isn't entirely unlike my manual coding process was anyhow.
The main problem is that pointing an LLM at a codebase and telling it to just "make it better" taps out pretty quickly. That is, not that there's zero juice to squeeze there, but there's not a ton. You can get a bit of improvement but it also rapidly starts changing things just to change things, which I'm not even going to complain about all that much because there's a sense in which it is simply doing as you asked.
So you still need human taste and direction. This will be especially true for something the size of that codebase where you can only hold small fractions of the actual code in the context window at once. Summaries only get you so far.