AI Is a Bad Tool

I can only imagine that people who say things like:

> If you use AI for anything else, and in particularly if you use it to generate code, you're wasting your time.

Have not used frontier models in at least a year.

It is nearly inconceivable to me that I would ever go back to writing code by hand, in any context. Even if no new model was ever released, the combination of GLM 5.2 and DeepSeek V4 flash is more than sufficient to do real work. It's not hard to imagine that a distillation of Mythos/5.6S or whatever is a couple generations away will push me even further in this direction.

> Recently, there's a lot of talk about AI allegedly finding security flaws in software. That is an unsubstantiated claim. As such, it would need to be verified by a non-machine, and arguably, the verification process would require the same amount of effort or more than would be required to find the issue to begin with.

This is simply not true. Security flaws are a great use-case for AI specifically because they're easy to verify. If you can drive a program to segfault based on inputs, you've got a good indicator it is, in fact, a security vulnerability (at minimum a DoS, but usually you find out later it was exploitable). You could even have the AI generate an exploit PoC. Shell? Valid hole. Done.

The bad use cases for AI are the ones where it's as or more expensive to verify correctness as it would have been to find the solution in advance.

This statement is just ridiculous:

> Recently, there's a lot of talk about AI allegedly finding security flaws in software. That is an unsubstantiated claim. As such, it would need to be verified by a non-machine, and arguably, the verification process would require the same amount of effort or more than would be required to find the issue to begin with.

Once a security issue is found, it is often far less effort to verify it. Digging through millions of lines of codes to find high-probability vulnerabilities is the hard part.

Pure sophistry, divorced from reality.

It starts with the assumption that nothing coming from AI can be useful or trusted and uses that to demonstrate that AI is not useful or safe.

Has the author used an LLM to brainstorm architecture, explore trade-offs, challenge assumptions, or refine a design? Models are not just 'a data distiller'. "How could I implement X feature in Y project?" requires systematic planning. The agentic paradigm is about balancing (expensive) human and (cheap) machine cognition. Frontier models chew through requirements and if you don't like what they come up with you can TALK to them about it.

I don't really agree with this. I was VERY skeptical about AI, but then I started using frontier models everyday, and my feelings have changed dramatically. I use AI to refine designs. I use AI for "where are all the codepaths where we return an HTTP 499 error". I use AI to write code (in "manually accept edits" mode, and I am picky about every single diff) and the results are really good. Most of the time, exactly what I typed in. Doing stuff like "add this config and plumb it through the helm chart" is amazing. Editing Go templates that generate YAML is absolutely miserable, it is my least favorite task. Claude gets it right without fuss every time.

I have not stopped engineering software, but I type in less code. I still understand every line in our codebase and why it's there, but I spend a lot less time dealing with the mechanics. LLMs are super impressive for the software engineering work I do. I don't have a fleet of agents making 20 major changes to the codebase every day. I do about as much work with AI as I did before I used AI. But I think the quality has gone way up, and I am a stickler for pretty perfect code. Claude keeps me honest and reduces the cost of exploring alternate approaches, writing hairy tests, doing refactors, keeping docs up to date, keeping production stable and understandable, etc.

I basically do not agree with this article at all. Frontier models are like switching from dabbrev-mode to IntelliSense, something I was very resistant to at the time. "It's important that I know the APIs that I use everyday." Not really. Having your IDE remember whether it's HasPrefix(string, substring) or HasPrefix(substring, string) frees up my brain for something more important and I couldn't live without it. AI is the same way. It's a tool that makes me better at my work. Yes, it's expensive. That's the only downside I see. I am lucky that I can use as much Fable 5 as I want at work... I always feel bad when I share something that it did that was cool with someone who can't afford it. But costs will come down. I think this AI think is here to stay in the software engineering space. I don't think it removes the need for qualified software engineers at the controls. At least today.

The idea that it only reveals missing abstractions and that this is bad, is pretty off the mark in a lot of ways.

First off, there's plenty of cases where it make no sense for me to spend all my time abstracting everything possible in my pipeline. Maybe I'm a game developer who doesn't want to spend all my time abstracting away all the initial scaffolding for game prototype and just make a dang game prototype.

Or maybe the abstraction would needs to much confirmation to not really be any different than code (again, games.)

Or maybe the LLM is indeed a perfectly good abstraction for such a task, reliable and customizable.

Just keep writing abstractions all the way up the chain until you're so abstracted that you take english input and can product whatever is asked for, and, oh right.

> The real question is, who can verify that what the AI built is good and true? Recently, there's a lot of talk about AI allegedly finding security flaws in software. That is an unsubstantiated claim. As such, it would need to be verified by a non-machine, and arguably, the verification process would require the same amount of effort or more than would be required to find the issue to begin with.

Came for the bad AI take, stayed for the P=NP claim.

This article starts off by saying AI can be useful for question-answering, but then quickly states that if you're using AI to generate code "you're wasting your time:"

> The reason AI is a bad tool is that generally speaking, it is completely opaque.

The author is missing the fact that you can use those great question-answering and knowledge distillation capabilities to ask the AI questions about the code it wrote.

Later, they write:

> Who is going to verify that it is doing what it is supposed to?

What? If you hire a writer to write a blog post, who is going to verify that they didn't just fill in gibberish? It's you. You there, in the front. You're the one.

If you can't find any way to independently verify that the software is doing what you think it should do, I don't think you have any business generating it, whether you use AI or code it by hand. That's like commissioning a portrait of someone you've never seen.

Is this satire? That's my most charitable interpretation.

This smells like it was written by someone who doesn't actually write code. How does one make the logic leap that any code AI is able to write, is only because that code is "trivial"? The author doesn't even bother defending that claim.

I suppose the only other charitable interpretation would be to say, "Well then 99% of the code I'm paid to write is 'trivial'." If that's the case, then you have to separate "code as means to an end" (which is clearly well within the wheelhouse of AI) from "code as a mathematical art" which is okay, but the latter doesn't pay my bills or create value for my clients.

In this article you can learn that it takes the same amount of effort to verify if a security hole is real, as it takes to find it in the first place. (?)

This article feels weirdly anachronistic with talking points I haven't really heard anyone mention since like 2023.

I'd welcome someone pointing out where I'm wrong in my critique of this critique..

"The reason AI is a bad tool is that generally speaking, it is completely opaque."

"The real question is, who can verify that what the AI built is good and true?"

Well, you? The developer? The person responsible for using the tool, no?

Isn't this article just saying, "using a tool blindly and not checking the result" is bad? Which, of course, it would be.

The AI posts will continue until morale improves.

Honestly, I am starting to feel it’s Groundhog Day every day here, reaching the same discussion we’ve been having for years now.

Condescending, ignorant, and the author doesn't know what he is talking about.

I'll not explain why, and that would be as deep as the author explanation of his coping.

Why those articles get to hacker news?

Yes.

AI (in its current state) is phenomenal for research on just about any topic; you can dive deep and cross-reference the material more easily than ever.

AI (in its current state) is not so good at actually doing things; we're a long ways off from simply speaking software into existence.

I get their point, but they are probably wrong. Not saying the AIs will not get better and better. They probably will, but I see a chance that there are smaller and smaller gains and the AIs cant make those gains happen. But coding with llm's doesnt say its shoddy code per se. I view it as: We used to use hand saws to make cabinets. Then we had tracksaws. Then we had CNC saws... You still need to understand woodworking to make a nice table in all these scenarios.

> Maybe being passionate about something you build, understanding it fully, owning it completely, and abstracting it properly

This is the actual job description. It doesn't matter whether AI is used for writing the literal code.

All that can really be said is that AI alone is inadequate for this job. None of this back and forth has ever really been about AI.

So, can we just cut the crap already and go back to the clarity we used to have on this? Bad leadership leads to bad outcomes. Bad leadership wants to use any tool at their disposal to micromanage and cut costs. Bad leadership is terrified of a younger and more capable workforce that will replace them. Bad leadership will do everything to deflect blame including calling you "out of touch" for not using AI enough or at all.

Not all workplaces are like this. I am actually glad AI has put a massive spotlight on these longstanding tensions.

Aging is happening faster than AI. It's far more likely there will be a time soon when all these software devs will no longer code because they're management (or burned out and changed careers). It's more important to start thinking about your integrity now before your lack of it gets you fired in the future anyway. I don't blame young people for being so naive, but this is a blindspot. Get a grip now.

Great article. Yes, often what's needed are better (rock-solid) abstractions that we can manipulate around quickly and confidently (think Abelson/Sussman SICP style), not a machine that can take good guesses at piles of spaghetti code.