When the model write new code doing the same thing as existing logic that's not a refactor.

At times even when a function is right there doing exactly what's needed.

Worse, when it modifies a function that exists, supposedly maintaining its behavior, but breaks for other use cases. Good try I guess.

Worst. Changing state across classes not realising the side effect. Deadlock, or plain bugs.

When they decide to touch something as they go, they often don't improve it. Not what I would call "refactoring" but rather a yank of the slot machine's arm.

> In practice, it was seldom done, and here we have LLMs actually doing it, and we're realising the drawbacks.

I spent some time dealing with this today. The real issue for me, though, was that the refactors the agent did were bad. I only wanted it to stop making those changes so I could give it more explicit changes on what to fix and how.

So I think theres some more nuance than that. A lot of the times, the abstraction is solid enough for you to work with that code area, ie tracking down some bug or extending a functionality. But sometimes you find yourself at a crossroad - which is either hacking around the existing implementation, or rethink it. With LLMs, how do you even rethink it? Does it even matter to rethink it? And on any who, those decisions are hidden away from you.

That's a real question, maybe the changes are useful, though I think I'd like to see some examples. I do not trust cognitive complexity metrics, but it is a little interesting that the changes seem to reliably increase cognitive complexity.

Really? I've never heard it's considered wise to put refactoring and new features (or bugfixes) in the same commit. Everyone I know from every place I've seen consider it bad. From harmful to a straight rejection in code review.

"Refactor-as-you-go" means to refactor right after you add features / fix bugs, not like what the agent does in this article.

There is a pretty substantial difference between "making changes" and "refactoring"

If LLMs are doing sensible and necessary refactors as they go then great

I have basically zero confidence that is actually the case though

>The idea being that if you're working in an area, you should refactor and tidy it up and clean up "tech debt" while there.

This is horrible practice, and very typical junior behavior that needs to be corrected against. Unless you wrote it, Chesterton's Fence applies; you need to think deeply for a long time about why that code exists as it does, and that's not part of your current task. Nothing worse than dealing with a 1000 line PR opened for a small UI fix because the code needed to be "cleaned up".