alt Hacker NewsI'm sure this method has evolved and/or been supplanted over the last 15 years, but one thing that struck me reading this is how much the dynamics of unit test coverage have changed in recent history, with AI-generated commits containing 10x as many unit tests (many of them kind of silly and tautological) as in the olden days. Gonna need to update some of those coefficients in their CRAP1 formula... Or maybe test coverage has/will become too noisy a parameter to use at all.
Replies
> Or maybe test coverage has/will become too noisy a parameter to use at all.
It already is, ive banned unit tests via ci checks from our codebases, they were not particularly useful before LLMs and now they are a net negative.
We require int and some e2es and that does all that units do and more.
[dead]
Anecdotally, I’ve found that codebases that enforce code coverage metrics often have worse behavior coverage than ones that don’t.
It’s a classic example of Goodhart’s Law in action. Code coverage metrics only measure what percentage of code the test suite causes to run. But it’s very, very easy to write tests that run code without actually confirming that it produces correct output for all possible inputs. And it’s very, very easy to assume that a module with 90+% code coverage also has 90+% behavior coverage, and then become complacent about reviewing the suite for proper behavior coverage.