alt Hacker NewsNo evidence most of the activities actually save money with modern ways of delivering software (or even ancient ways of delivering software; I looked back and the IBM study showing increasing costs for finding bugs later in the pipeline was actually made up data!)
Replies
Last rant: everybody is testing in production, but only some people are looking at the results. If you aren't then there's better ROI to be found than "shifting left" your e2e tests.
Are you referring to the IBM Systems Science claims (likely apocryphal) in the Pressman paper, or Barry Boehm's figure in "Software Engineering" 1976 paper which did include some IBM sourcing (literally drawn on) but was primarily based on survey data from within TRW?
It baffles me that anyone would continue to promulgate the Pressman numbers (which claim ~exponential growth in cost) based on... it's not entirely clear what data, as opposed to Boehm's paper which only claims a linear relative cost increase, but is far more credible.
To be more specific, let's say I can write an e2e test on an actual pre-prod environment, or I can invest much development and ongoing maintenance to develop stub responses so that the test can run before submit in a partial system. How much is "shifting left" worth versus investing in speeding up the deployment pipeline and fast flag rollout and monitoring?
Nobody I've worked with can ever quantify the ROI for elaborate take test environments, but somebody made an okr so there you go. Far be it we follow actual research done on modern software... http://dora.dev