alt Hacker NewsI agree. The latest models are getting quite good at validation, as long as they are clear on what the requirements are. Recently I got frustrated that while we have a huge test suite I noticed the LLM was hardly ever breaking a test with its changes (PRs only added tests, never changed old ones) which seemed suspicious.
I had it do a full audit. There were about 100k lines of tests, of which it determined about 60k were near useless. Many were overly mocked, some mocked to the point that they really weren't testing anything at all. It deleted those and wrote more expansive scenario tests to replace them, caught a lot of bugs on the way and fixed those too. Total time was about 2 hours.
I think humans need to review requirements, specifications, and test scenarios. And for some industries and uses they're still going to have to review every detail. But for a typical CRUD app, I think we're at a point where we can already set the process and quality bar higher than ever, in addition to moving faster.
Replies
Glad to see someone else notice this. I dread looking at or dealing with AI written unit tests because of the incredible amount of whiteboxing that goes on, though usually for the opposite reason because AI testing is almost entirely whitebox testing which fails if some internal details change
Yes, unfortunately a lot of requirements and specifications are now AI slop, thus developers stop reading them, and agent produce even more slop.
We need to take back writing requirements and specifications to humans.
Adopting high level BDD style testing of human written user stories (cucumber) is probably a good idea.
I chuckle a little bit at this specification thing. We used to do this. We used to write big binders of specifications, and then we'd set about implementing them. We stopped doing it because it didn't really work. We'd learn things in the process of building which negated the value of the big specs we wrote. That's why we stopped doing it. This is where continuous improvement, CI/CD, tighter release cycles, etc. were all born.