logoalt Hacker News

jerf • today at 5:38 PM • 0 replies • view on HN

I think the problem with testing vocabulary is that the field is just too continuous. We have all sorts of situations in the world where there is no sharp line, yet we have terminology for various chunks of the possibility space even so because there are still some sort of clump in the space that we can identify. For example, there isn't a true sharp line between any of the primary colors, but we do not generally have a problem labelling things as "red" or "green" or whatever even so. There is a certain personality type frequently encountered online that this bothers no end, who can't really deal with the idea of words that have fuzzy boundaries around them and struggle with the concept of a "fuzzy line", but most of us get on even so.

However, when there is no natural clumping, there isn't even a place to put the fuzzy boundary necessarily. I think a lot of testing has this problem. I've disliked "unit" versus "integration" terminology for at least a decade now, because to me there's just a continuum. Yes, obviously, tests can test more or less of the code base at a time, and can draw in fewer or more dependencies in the process, and arguably there is an important distinction between 0 other dependencies included and 1, but beyond that I've thought for a long time there aren't any other distinctions worth putting a label on.

There is value to tests that mock out all the other dependencies. There is value to writing stubs. There is value to bringing up those dependencies and mocking or stubbing out their dependencies. There is value to having a full staging environment and running acceptance tests by hand, and that staging environment itself can have any number of mocks or stubs in it itself. And there is no rule that says you rigidly must make this decision based on the exact number of dependencies away something is from your code, to the extent that that is even a well-defined concept, nor are "mock" and "stub" themselves some sort of bright shining line as one can mix and match them freely, nor is that the complete list of all possibilities. The result is a fairly smooth continuum of possible tests. In my opinion, any attempt to put labels on very particular bits of it raises the risk of overprivileging certain parts of the space just because they get a name and creating the streetlight effect on the named bits and causing the others to be missed.

Even "performance testing" versus "correctness testing" gets awfully fuzzy; again, there is arguably some value in saying this test has performance concerns and this other test is just related to correctness, but even "performance testing" still has to worry about a lot of other things too and I don't find it anywhere near a binary distinction. Anyone who has spent any time with performance testing has certainly learned the risk of worrying about "performance" without also worrying about all the other things we test for at the same time, or to put it another way, I can make my performance testing look really good if I let the code it is testing be incorrect.

It is unreasonable to expect a universal nomenclature catalog of such a smooth space. We can't even as a species completely agree on color delineations in the spectrum, which is a simpler problem in most ways.