alt Hacker News> We don’t yet know how to keep quality high and complexity low while doing agent coding, we’re just having them write lots of tests and hoping that when things break they’re easy to fix
Implement more versions with different languages, frameworks, libraries and AI agents. Then you can do differential execution for testing, which is easier than imagining testing scenarios from pure code. You can fuzz or use production data (shadow mode execution) to get notified early of divergences.
Keeping a population of diverse replicas might be the simplest way to derisk AI code. When 2 replicas diverge an agent can decide - was it a bug in one of them, or was the spec unclear? Either way, if you fix a bug or clarify the spec you come ahead.
A separate class of problems is related to intent divergence. For that my solution is to keep a full log of all user messages during the lifetime of the project. Even 10,000 messages collected over weeks and months can be read and summarized in a few minutes. That means you can check for intent divergence with agents, but you always start from the raw source (your human messages) not from intermediate summaries and diverged code.
Combining the 2 ideas would make each replica agent derive the user intent separately from the same log.
This is a nice idea, but I’m in a big enough and low-level enough code base that using other languages or stacks isn’t very viable; we build the stack, so it’s software and hardware and compiler and driver. We do sometimes use multiple agents and adversarial review. The agents tend to come up with more complete corner-case testing than humans, the most common issue is that they don’t fully understand all design assumptions and parameters and they over-test for things that can’t happen using very complicated test setups, and they don’t test enough for simple mainline usage; they generally don’t know what the common workflows are. I had the same complaint about humans before LLMs too. :P We often try to be clever about testing and design all this logical coverage while ignoring the boring & easy stuff, we think like engineers and not users.