alt Hacker NewsLike becoming a manager?
When a person becomes a manager, they do or do not have enough time and expertise to review all of the code that they trust the team to produce.
Managers usually get into automated testing; unit tests, integration tests, acceptance tests, and maybe also BDD syntax
Managers and developers are responsible for setting a test coverage threshold for merge approval.
If there is 100% branch coverage test coverage for a codebase, what would coverage-guided fuzzing or property testing find? If there is 100% branch coverage test coverage for a codebase, what is the value of spending resources on formal verification?
How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage?
Replies
Formal verification is always more valuable than mere testing, but it's hitherto more expensive. The thing that ultimately matters is closing the loop: how well do the tests match the requirements, both as written and as unwritten in the mind of the customer?
Working with an LLM has given me a real eye opener on unwritten requirements. It's like outsourcing. "Yes, you've given me what I wrote down, but I never expected you do to it in that way"
> When a person becomes a manager, they do or do not have enough time and expertise to review all of the code that they trust the team to produce.
> Managers usually get into automated testing; unit tests, integration tests, acceptance tests, and maybe also BDD syntax
I can see managers getting involved into acceptance tests, but never in the other type of tests. And the verification mostly is involved into a quick manual testing/watching a demo. Code is not their concern. When there's a bug, they expect you to investigate and fix it.
I think what this fails to cover is that managers rely on other people to do that understanding and keeping things in mind. When there's a problem, the manager isn't just saying to a random person "go fix this", most of the time they're relying on somebody who has a deeper understanding of the problem and is not likely to randomly break something else just to fix this one bug.
> How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage?
This is such a salient question. Sometimes (definitely not always) the test suites produced by LLMs are so trivial it's scary. Coverage can be an illusion for sure.