alt Hacker NewsI’d go further and say while testing is necessary, it is not sufficient. You have to understand the code and convince yourself that it is logically correct under all relevant circumstances, by reasoning over the code.
Testing only “proves” correctness for the specific state, environment, configuration, and inputs the code was tested with. In practice that only tests a tiny portion of possible circumstances, and omits all kinds of edge and non-edge cases.
Replies
Came to leave the same comment. It’s very possible to deliver code that’s proven to work, that is still shit.
I find myself not really trusting just tests, I really need to try the app/new function in multiple ways with the goal of breaking it. In that process I may not break it but I will notice something that might break, so I rewrite it better
Since we can't really formally prove most code, I think property based testing such as with hypothesis[1] would make sense. I have not used it yet, but am about to for stuff that really needs to work.
I agree - it's trivial to write 100% test coverage if your code isn't robust and resilient and just does "happy path" type stuff.
if your tests cover the acceptance criteria as defined in the ticket, why is all htat other stuff necessary?
Testing is not perfect, but what else is there? Even formal proofs are just another expression of testing. With greater mathematical guarantees than other expressions, granted, but still testing all the same; prone to all the very same human problems testing is burdened with.
I'd go further and say vibe coding it up, testing the green case, and deploying it straight into the testing environment is good enough.
The rest we can figure out during testing, or maybe you even have users willing to beta-test for you.
This way, while you're still on the understanding part and reasoning over the code, your competitor already shipped ten features, most of them working.
Ok, that was a provocative scenario. Still, nowadays I am not sure you even have to understand the code anymore. Maybe having a reasonable belief that it does work will be sufficient in some circumstances.
I would like to challenge this claim. I think LLMs are maybe accurate enough that we don't need to check every line and remember everything. High level design is enough.
> "proves"
I like using the word "demonstrates" in almost every case where people currently use the word "proves".
A test is a demonstration of the code working in a specific case. It is a piece of evidence, but not a general proof.
And these kinds of narrow ad-hoc proofs are fine! Usually adequate.
To rephrase the title of TFA, we must deliver code that is demonstrated to work.