I am more and more just using LLM assistance over fully agentic programming. Even when writing detailed instructions, guardrails, and whatnot, a lot of modern models (e.g. Opus) are optimized to work autonomously. But over time they just make a mess out of code bases, often because they don't truly understand good design, but will just make changes until they reaches their objective, leaving a mess for a human to clean up. Also, models like Opus always want to do too much work/research. I'm not sure if this is caused by Anthropic optimizing for tokenmaxing or the model being overoptimized for agentic programming.

I also notice this in PRs from of others. Yes, in a shallow sense, it solved the problem that they had in their prompt. But while doing it, their LLMs made very suboptimal solutions that are hard to maintain long-term.

So, I have increasingly resorted to smaller open source models or 'weaker' models like Claude 4.5 or 4.6, since they are often much better for LLM-assisted programming, where you have the implementation steps in your head (you know what the implementation should look like) and work towards the solution with the model step by step. Working like that is extremely powerful and much faster than programming by hand (an LLM memoizes/searches the APIs faster/better and can more quickly write the boilerplate that programming sometimes also requires. I feel like a can be a few times faster than my old self, while not leaving the code base behind in a mess, and not atrophying knowledge, since I still do most of the thinking, design, etc.

There’s been infinitely times where I was stuck at some problem and every single solution was complex, messy, over-engineered and somehow wrong, until I went on a walk or moved to a different issue and suddenly it would hit me that I was looking at it all wrong, and there’s a simple solution there but my tunnel vision didn’t let me see it. LLM rob you of that, everything is instant, there’s no time to reflect, there’s no time to realise it’s a dead end, or that it won’t work with the next problem on your todo list.

This has happened to my side project too. There's so much going on in the code now that I can't possibly grok it to make changes safely by myself any more. But I think that's OK. Because I don't have to grok it anymore.

I've been crafting beautiful code for almost 20 years - that itch is scratched. Now I just want to be productive and build cool stuff. AI is helping me make a better end-product in a fraction of the time. As long as codex understands what's going on in the ball of spaghetti, there's no problem. Sure - if I was working in enterprise then this is no good. But for personal projects and small indie shops, it's fine.

AI is just a new layer on top of the stack now. In the same way high-level languages are a layer on top of writing machine code manually. You just need to let go.

The real value I think never really was the code, it was the learnings from the journey to get to the code. The struggle allowed for iterative discovery of the real requirements and technical challenges. With AI you skip that path and it looks like you've achieved your goal only on a superficial level.

I think this doesn't mean AI is useless (it's not), it's just that we have to think harder about the requirements and end-stage verification and have less trust in the process to result into valuable outcome.

For my projects, I can produce with LLMs a codebase that I'd be proud of, but it requires extensive use of the plan mode and nagging anytime the implementation looks much more complex than the feature at hand requires.

Anytime I tried spec driven development, it produced total garbage, the problem is that the model usually swings too far, if they write unnecessary complexity, a nag about it and you risk it code golfing, this problem across 100 lines of specs and you're guaranteed it will swing too far on some segments you just wanted X slightly more than Y (usually for me it was reliability dropped for better readability since # users = # developers = 1)

Another thing I dropped is steering implementation design too early, start with the goal and nag in the direction you want, it works well for me but might not work well for big complex codebases.

For now I love LLM coding but I'm sure my opinion will change if I have to review code from a developer that does the 1 prompt = 1 PR without looking at the code jutsu.

I've found that "build X with framework Y" produces better results than "build X". Possibly because many architectural decisions have already been made by the framework authors.

Before you throw away, I propose having an LLM write a comprehensive spec from your code, then a clean-room implementation. It's an interesting experiment, if nothing else.

It’s a garbage collection problem. They’re good at adding stuff. They’re bad at removing stuff. You have to go out of your way to have them remove stuff; rubrics, byte for byte identical outputs, bug for bug compatibility etc and then convince the cursed things that it’s still safe to rip and replace. Bonus points when you get them to stop before the ‘replace’ and everything still works.

I’ve found it very useful to leverage the “speed” of AI to iron out all of the UX-questions of a greenfield side project (an organiser for a/my family).

My first aim is to get to what feels like v1 feature-completeness. My approach is building out the feature, dogfooding it for a few days, work out the bits I don’t like, adjusting, and then rinse and repeating.

I don’t like the finer UI details (the colours it’s chosen for instance), nor do I fully jive with how it’s implemented the business logic (it’s verbose and needlessly fancy in some places) but - that’s not what I’m concerned about now. I’m concerned about whether my wife and I find the tool useful. (We do, huzzah).

I’m now looking forward to sitting down and redesigning the finer details of the UI to my taste, and then, having established pretty solidly how the backend is going to need to work, how I would do it, and then crack on and do a rewrite from the ground up so that I can maintain/extend it more effectively going forward.

Which is all to say, I find Claude and it’s kin are excellent for prototyping (answering questions, giving someone something to play with), and for uncovering/discovering the requirements of a project which make it then easier to do “properly” if you feel it hasn’t done a good job already.

Everyone will have their own approach of course, but I’ve found it helpful to take this one in this case.

When you're in the trenches actually making code changes yourself, you get irritated and are driven towards improving the code as you implement new features. AI never does because it never has to. And because you're not in the trenches, you don't actually experience the frustration of managing the code base. So from your tactical view everything is fine, and when you're reviewing an isolated change you don't have the context of everything the AI had to read and figure out to get there. If you had to wade through shit to figure out how to implement something, you'd have made those refactors a long time ago. AI just eats it up and serves you a new plate of slop. It's a blessing and a curse depending on who you ask.

AIs greatest strength is still throwaway code.

The exact opposite has happened for me. I used to review every pr, give read only access to db. Now with codex 5.6 sol high I let it rip. I do sometimes quickly scan the pr but mostly I've been regulated to a manual tester. Run the feature, check the DB, verify the UI. Build unit and E2E around everything. Mind you I am not working with production data yet.

My stack for this project is React Native and Supabase which I've got a good amount of experience using.

Which LLM are you using?

It cannot write some kind of short texts either.

The experiment was: given existing Experience section of Senior Software Engineer CV (manually and carefully written), write 2-4 lines of high pitch About [me].

gpt-sol-xhigh just could not do it, making a complete AI slop mess.

It became substantially better when I asked to get a sample of real Senior/Principal engineers CVs, that it would be able to connect with meaningful online presence/contributions of their authors, and draw ideas from that.

Nevertheless, the remaining issues were critical, with their classes spanning:

- word for word repetition;

- tautology (phrases mapping to the same semantic entity);

- category mistakes (combining apples with oranges);

- faulty composition of generalised and concrete terms.

Before throwing it away I decided to give it a try and asked for strict prompt following, setting low logical errors threshold, and eventually providing a formal proof that it complied.

It took spaCy, doing NER and dependency parsing; then I suggested adding stanza for constituency parsing.

After slapping together the artefacts of analysis and thinking a bit, it produced great phrase (to my taste).

-

Then when I asked to apply the framework and improve bullet points in some experience block, it did 2 of 4 well, then miserably broke down; I’m not sure if it’s harness issue (codex) or fundamental model restriction.

-

So no, without substantial investment in steering, SOTA AI doesn’t perform even remotely close to a human, in complex reasoning.

> but when running multiple "Please look for issues" iterations it would never have spotted the issue by itself.

Nah that's old meta, you're supposed to generate candidate issues with an agent swarm running at high temperature. It's the only way to fly!

It's hard to take criticism like this seriously when you didn't say the model you used. It's like someone complaining about a car's performance but they leave out the brand and year. If you don't think that's relevant I just can't take you seriously.

You get a sense for when things start to drift.

“Any organization, cleanup or consolidation needed?”

This mirrors my experience too, long term use starts showing incoherence that is impossible to see in isolation.

The funny thing with LLM’s is that with a sufficient sized code base they pretty much will loop forever if you tell them to find and fix issues, making new ones as they fix old ones.

I don't have the same experience at work.

After two years of investment LLMs can write code mostly on their own due to a combination of proper architecture (you can't really "cheat it"), documentation, instructions and human checks.

On my personal projects, it's still a super good match for building prototypes, but at some point if you don't write it your own you lose the plot. So you're bound to write it your own before handing it off to LLMs.

In any case there are no black or white solutions, and there are aspects that make me think that the more capable the less fit it is for serious projects at some point.

In my experience AI doesn't create the same abstractions as a human developer and you see this very quickly on a complex code base if you let the AI run wild.

Humans, and especially good developers, naturally create really good mental models for how to think about complex systems. We have to. That's ultimately how we build complex systems.

Refactoring (when it's not simply upgrading packages or changing languages) typically happens because we realise there's a better way to model the system we're building and we want our code to reflect the model in our minds.

AIs today simply don't do this. They write code that solves a specific problem, and while they do this well, they don't seem to create well defined and well reasoned models about how to think about complex systems. Or maybe they do, but the AI definitely doesn't say, "hey, I think what you're building is more like x than y, so I want to spend 20 minutes of refactoring your codebase".

I guess to use an analogy, imagine trying to tell someone how to build a bike who has never heard of a bike before, and therefore has no mental model of what a bike is. You'd probably start by telling them to put two wheels on a frame, then to add a seat on top, then to a add some pedals and attachment them to the wheel with a chain... At the end of this process that person might create something bike-like, and it might kinda do the thing you want, but it would likely be very different from the bike someone would build if they had a very clear mental model of what a bike is.

I suspect this is why current AIs rapidly hit a limit after the prototyping stage. I don't think task duration is the right way to think about this limitation. I suspect it's more of a proxy for how complex a task can become before the AI isn't able to create a deep enough model of the problem it's trying to solve, so begins to hit complexity limits and just spits out spaghetti code.

AIs are getting better quick though. I think they'll get there soon, but people are creating a lot of mess in their codebases in the mean time.

LLMs can write excellent code a few thousand lines at a time but they absolutely can't create a cohesive million line codebase.

I had a funny yet tragic experience about LLM last week

A CICD flow had broken. It was not giving the URL of a deployed service.

A team member used AI to fix it without reading the code and AI added 55 lines of code

When I noticed the issue, I modified the code by removing 5 extra lines and used the right command to get the url

Earlier I was doing a grep and matching the output against .com to get the url Recently I noticed that the utility provided a --format json option that was much cleaner!

> but looking at the whole picture everything is subtly wrong in multiple ways.

That would also make a great description for human societies, wouldn't it?

Actually, after second thought, actually LLMs at least give the surface level impression, which might not be exactly the case for humanity at the moment.

Ok, sarcasm apart, I'm fine with LLM at the moment as a technology. How we use it, what impact it has on society is certainly more of a concern than the technology itself. Compared to weapons, at least they're not fundamentally designed specifically to harm and threat.

[flagged]

[flagged]