Dev-Owned Testing: Why It Fails in Practice and Succeeds in Theory

The conversation is usually: devs can write their own tests. We don't need QA.

And the first part is true. We can. But that's not why we have (had) QA.

First: it's not the best use of our time. I believe dev and QA are separate skillset. Of course there is overlap.

Second, and most important: it's a separate person, an additional person who can question the ticket, and who can question my translation of the ticket into software.

And lastly: they don't suffer from the curse of knowledge on how I implemented the ticket.

I miss my QA colleagues. When I joined my current employer there were 8 or so. Initially I was afraid to give them my work, afraid of bad feedback.

Never have I met such graceful people who took the time in understanding something, and talking to me to figure out where there was a mismatch.

And then they were deemed not needed.

Oh man do I have opinions.

First of all, I've seen all type of teams be successful, ranging from zero QA at all, to massive QA teams with incredible power (eg. Format QA at Sony in Europe). I have absolutely seen teams with no QA deliver high quality full stop, the title is nonsense.

My firm belief is that QA can raise the ceiling of quality significantly if you know what you're doing, but there is also huge moral hazard of engineers dropping the ball on quality at implementation time and creating a situation where adding more QA resources doesn't actually improve quality, just communication churn and ticket activity. By the way the same phenomenon can happen with product people as well (and I've also seen teams without product managers do better than teams with them in certain circumstantes).

The most important anchor point for me is that engineering must fundamentally own quality. This is because we are closer to the implementation and can anticipate more failure modes than anyone else. That doesn't mean other roles don't contribute significantly to quality (product, design, QA, ops absolutely do), but it means we can't abdicate our responsibility to deliver high quality code and systems by leaning on some other function and getting lazy about how we ensure we are building right.

What level of testing is appropriate for engineers to do is quite project and product specific, but it is definitely greater than zero. This goes double in the age of AI.

Most orgs I've worked for are so growth and product-focused that if you try adjusting your estimates to include proper testing, you get push back, and you have to ARGUE your case as to why a feature will take two weeks instead of one.

This is the thing I hate the most about work, having to ARGUE with PMs because they can't accept an estimate, there's often some back-and-forth. "What if you do X instead?" "Team Y (always uses hacks and adds technical debt with every single feature they touch) did something similar in two days." But we're just communicating and adding transparency so that's good and it certainly doesn't matter that it starts taking up 4+ hours of your time in Slack conversations and meetings of people 'level setting' 'getting on the same page' trying to help you 'figure out' how to 'reduce scope' etc. etc.

Also, I think testing via unit or integration tests should be standard regardless, and that isn't what I am thinking about here. I'm thinking about QA, the way QA does it. You hammer your feature with a bunch of weird bullshit like false and unexpected inputs, what happens if I refresh the page in strange ways, what happens if i make an update and force the cache to NOT clear, what happens if I drop my laptop in a dumpster while making the request from firefox and safari at the same time logged in as the same user, what happens if I turn off my internet in the middle of a file upload, and so on. When devs say that devs should be responsible for testing, they usually mean the former (unit and integration tests), and not this separate skillset of coming up with a bunch of weird edge cases for your code. And yes, unit tests SHOULD hit the edge cases, but QA is just better at it. You usually don't have engineers testing what happens when you try sending in Mandarin characters as input (unless they live in China, I guess). All of that effort should bring up your estimates because it is non-trivial. This is what getting rid of QA means, not happy path end-to-end testing plus some unit and integration tests.

While good points are made, I worry this gives the wrong impression. The paper doesn't say it is impossible, just hard. I have, very successfully, worked with dev owned testing.

Why it worked: the team set the timelines for delivery of software, the team built their acceptance and integration tests based on system inputs and outputs based on the edges of their systems, the team owned being on-call, and the team automated as much as possible (no repeatable manual testing aside from sanity checks on first release).

There was no QA person or team, but there was a quality focused dev on the team whose role was to ensure others kept the testing bar high. They ensured logs, metrics, and tests met the team bar. This role rotated.

There was a ci/cd team. They made sure the test system worked, but teams maintained their own ci configuration. We used buildkite, so each project had its own buildkite.yml.

The team was expected by eng leaders to set up basic testing before development. In one case, our team had to spend several sprints setting up generators to make the expected inputs and sinks to capture output. This was a flagship project and lots of future development was expected. It very much paid off.

Our test approach was very much "slow is smooth and smooth is fast." We would deploy multiple times a day. Tests were 10 or so minutes and very comprehensive. If a bug got out, tests were updated. The tests were very reliable because the team prioritized them. Eventually people stopped even manually verifying their code because if the tests were green, you _knew_ it worked.

Beyond our team, into the wider system, there was a light weight acceptance test setup and the team registered tests there, usually one per feature. This was the most brittle part because a failed test could be because another team or a system failure. But guess what? That is the same as production if not more noisy. So we had the same level of logging, metrics, and alerts (limited to business hours). Good logs would tell you immediately what was wrong. Automated alerts generally alerted the right team, and that team was responsible for a quick response.

If a team was dropping the ball on system stability, that reflected bad on the team and they were to prioritize stability. It worked.

Hands down the best dev org I have part of.

Something I've always believed, and my experience with shipping multinational software on a schedule that has severe drop-dead dates confirmed: If you are contractually obligated to deliver a product that does x, y, and z correctly? QA is the only way to do that seriously. If you don't have QA, you don't care about testing full stop.

This only compounds when you have to comply with safety regulations in every country, completely setting aside the strong moral obligation you should feel to ensure you go far above & beyond mere compliance given the potential for harm. This compounds again when you are reliant upon deliverables from multiple tiers of hardware and software suppliers, each contract with its own drop-dead dates you must enforce. When one of them misfires, and that is a "when, not if", they are going to lie through their teeth and you will need hard proof.

These are not small fines, they are company-killing amounts of money. Nobody profits in this situation. I've been through it twice, both times it was a herculean effort to break even. Hell, even a single near-miss handled poorly is enough to lose out on millions in potential future work. The upsides are quite nice, though. I didn't know it was possible to get more than 100% of your salary as a bonus until then.

Don't take my word for it, though. Ask your insurance agent about the premiums for contractual liability insurance with and without a QA team. If you can provide metrics on their performance, -10-15% is not uncommon, this discount increases over time. Without one? +15-50% depending.

If as a developer you want to be seen as someone advancing and taking ownership and responsibility, testing must be part of the process. Sending an untested product or a product that you as a software engineer do not monitor, essentially means you can never be sure you created an actual correct product. That is no engineering. If the org guidelines prevent it, some cultural piece prevents it.

Adding QA outside, which tests software regularly using different approaches, finding intersections etc. is a different topic. Both are necessary.

A nice piece that outlines all the challenges, the opportunities, and the cultural and social adjustments that need to be made within organizations to maximize the chance of left-shifted testing being successful.

IMPO, as a developer, I see QA's role as being "auditors" with a mandate to set the guidelines, understand the process, and assess the outcomes. I'm wary of the foxes being completely responsible for guarding the hen-house unless the processes are structured and audited in a fundamentally different way. That takes fundamental organizational change.

Developers want things to work.

QA wants things to break.

What worked for me, devs write ALL the tests, QA does selective code reviews of those tests making devs write better tests.

I also wrote the failure of Dev-Owned Testing: "Tests are bad for developers" https://www.amazingcto.com/tests-are-bad-for-developers/

The abstract says it really:

"It was clearly a top-down decision"

Many many things that are imposed like this will fail.

Its not willful non-compliance even, its just that its hard for people to do things differently, while still being the same people in the same teams, making the same products, with the same timelines...

Context is key here, lots of people see a thing that works well and think they can copy the activities of the successful team, without realising they need to align the mindset.. and the activities will follow. The activities might be different, and thats OK! In a different context, you'd expect that.

I'd argue that in most contexts you don't need a QA team at all, and if you do have one, then it will look a lot different to what you might think. For example, it would be put after a release, not before it.. QA teams are too slow to deal with 2000+ releases a year - not their fault, they are human.. need to reframe the value statement.

As a developer, I frequently tell higher ups that "I have a conflict of interest" when it comes to testing. Even though I fully attempt to make perfect software, often I have blind spots or assumptions that an independent tester finds.

That being said: Depending on what you're making and what platform(s) you target, developer-owned testing either is feasible or not. For example, if you're making a cross-platform product, it's not feasible for a developer to regression test on Windows 10, 11, MacOS, 10 distros of Linux. In contrast, if you're targeting a web API, it's feasible for a developer to write tests at the HTTP layer against a real database.

This paper has 7 references and 4 of them are to a single google blog post that treats test flakiness as an unavoidable fact of life rather than a class of bug which can and should be fixed.

Aside from the red flag of one blog post being >50% of all citations it is also the saddest blog post google ever put their name to.

There is very little of interest in this paper.

My experience with this was great. It went really well. We also did our own ops with in a small boundary of systems organized based on domain. I felt total ownership for it, could fix anything in it, deploy anything with any release strategy, monitor anything, and because of that had very little anxiety about being on call for it, best environment I ever worked in.

> The problem is not that dev-owned testing is a flawed idea, but that it is usually poorly planned

In our case there was zero plan. One day they just let our entire QA team go. Literally no direction at all on how to deal with not having QA.

It's been close to a year and we're still trying to figure out how to keep things from going on fire.

For a while we were all testing each other's work. They're mad that this is slowing down our "velocity", and now they're pushing us to test our own work instead...

Testing your own work is the kind of thing an imbecile recommends. I tested it while I wrote it. I thought it was good. Done. I have all the blind spots I had when I wrote it "testing it" after the fact.

This just seems to be basically a blog post that somehow got published in ACM?

Do people actually send PRs with no tests? That is so bizarre to me

The article argues that Dev-Owned testing isn't wrong but all the arguments it presents support that it is.

I always understood shift-left as doing more tests earlier. That is pretty uncontroversial and where the article is still on the right track. It derails at the moment it equates shift-left with dev-owned testing - a common mistake.

You can have quality owned by QA specialists in every development cycle and it is something that consistently works.

Note: the references in the article seem to be incorrect [4] through [7] are all the same article. I do not think that was intentional.

First they came with the NoOps movement, and you were happy cause those damned ops people were always complaining and slowing you down. I can manage my infra!

Then, they came with the dev-owned testing and fired all the QAs, and you were happy because they were always breaking your app and slowing you down. I can write my tests!

Now, they are coming with LLM agents and you don't own the product...

The paper highlights the problem in two words of the first sentence of the abstract: "shrink QA".

Corporations do it to save money, and accept the loss of quality as the cost of doing business. Therein lies part of the reason for the sad state of software today.

Author-Owned proof reading is next

I think they've got that the other way around.