alt Hacker NewsIt's a losing proposition. Self-driving cars go from point A to point B, and it's really easy to specify point B. My experience so far using agentic AI 24/7 for almost a year has been the AI can drive almost anywhere, so the whole problem is specifying where to drive. This doesn't get solved with faster, better AI, because all that will do is allow it to autonomously drive your codebase to random places you didn't want it to go. Then you're still left with the problem of specifying the problem.
But then there's the cases where the AI can't actually drive. What if you tell the agent to invent AGI? Or time travel? Where does it drive to? Does it tell you that it can't? How does it know it can't just drive here? How you you know? At the end of the day these are not wish granting machines, so someone connected to reality is still going to have to make decisions. And that person is going to be the one held liable for whatever the AI does so would they want a self-driving codebase in the first place?
Well put. I've found that agents are great at adding features to areas of the code that I already understand, where I can phrase my goals coherently and keep them on the rails.
Any attempt to describe software by "specification" leaves enough ambiguity that the agent will do deranged stuff like add a ton of code to satisfy error cases that can't happen. Or, like you imply, it will confidently create a bunch of nonsense to "solve" a problem in a way that is not really possible.