alt Hacker News> generate an uncharacteristically exhaustive suite of unit tests to validate every possible scenario.
This is what you want. You want comprehensive tests at every level, far more than is reasonable for a human to build or maintain, from unit, functional, to full end to end and beyond. Adversarial testing (both TDD-style "write tests to demonstrate this bug", and posthoc "prove this patch wrong with a new test") is the best way to keep AI on track and make those diffs you have to read clean and easy.
An even better way is to use a more strongly typed language and really lock it down, but you can use testing in any language. I feel like my background in TDD and "TATFT" has been secret sauce when working with AI
Replies
I used an LLM to build this
https://github.com/dprkh/eventfs
It has good test coverage, mostly unit tests but also a number of end-to-end tests. I also made the LLM build a benchmark, which you can find at the bottom of the readme. It is obviously slow, but I thought that it is good enough to work. When I tried to write a 1 GiB file, I found that it broke down, and after writing half the file, the speed went to under one megabyte per second. Implementation is 10k+ LoC, and I have no idea what is going on there.
100% this is what I've done. I sucked it up and adapted myself to the tool (agents) by having as many implicit guardrails (static typing, functional, no nulls, great linting) and then layering on explicit guardrails (TDD) on top. I also want my workflow to be portable because I don't really trust the frontier model providers.
It is different though. Basically a lot of what I do has changed over the last 2 years. I totally get that a lot of people won't want to adapt though.
Yeap. Hundreds of tests for small tools are completely trivial now.
I see this get mentioned a lot but I still am skeptical that AI can generate tests we can trust more than any other code we know we cannot trust.
Yes tests are conceptually isolated and that helps, but I've personally seen unit tests get generated that are semantically incorrect - that is, they test the structure of the code (e.g. they can check function output types and values), but they can't know _why_ the unit tests need to be there, so the really really helpful tests never get generated. Not to mention the obvious issues with generated tests only testing is x = x, or needless redundant tests for the same thing, or them essentially testing basic features of the language.