logoalt Hacker News

stevenburg • today at 6:54 PM • 1 reply • view on HN

That part is the weird bit to me. If the existing harness breaks right away, that's the signal to stop and fix things, not to book an outside audit. Running the suite in CI on every build would have caught a lot of this.

Replies

shoo • today at 8:56 PM

> Running the suite in CI on every build would have caught a lot of this

yeah, provided:

  1. test cases are being designed to cover functionality
  2. test cases are implemented correctly - i.e. they actually fail if the implementation is defective
  3. the tests aren't all commented out
  4. the test suite fails if a test fails
  5. CI actually detects test suite failure
  6. failing tests block changes from being merged (apart from initial triage to stabilise the suite)
Presumably a few of these things weren't being done -- I'd bet 1 & 2 were skipped or implemented so poorly that they were worse than having no tests at all -- checking nothing but giving a false illusion that some kind of testing had happened.

Five years ago, outsourcing your software development (including QA) to the lowest bidder might have gotten you results like this -- if you didn't explicitly require automated tests as a deliverable, you wouldn't get any, or if you got some they'd be vacuous and not test anything.

➕ show 1 reply