A senior advocate I knew treated every confident claim the same way: “very good, now show me the record.” Software teams would benefit from the same instinct. The subject in the dock today is TestOps — the practice of running automated tests as a service whose judgments developers act on. Around it has grown a cloud of confident assertions, repeated in meetings until they harden into precedent. Claims, however, are not findings. Six of the most common are called to the stand below, where the published record gets to answer first.
Claim one: “TestOps means the tool will provide”
Counsel for the defence produces a brochure. The record replies: buying software is neither necessary nor sufficient. Practitioner surveys — the only peer-reviewed one, presented at the XP conference in 2020, found definitions of the term wildly inconsistent among the very people using it — and no certification turns a purchase into a practice. What tools do provide is plumbing: parallel execution, browser grids, throwaway databases, contract verification, performance thresholds. The discipline sits on top, in decisions about budgets, containment and honesty. A useful self-test separates the two instantly: if the tool were switched off tomorrow, would the team still know its false-alarm rate? Teams that can answer yes bought plumbing. Teams that cannot bought a hope. The art of comparing software before buying applies here with full force.
Claim two: “A test that fails sometimes is a bad test — remove it”
Summary execution, without inquiry. Cross-examined, the record shows that intermittence is a symptom with several possible diseases:
- races between steps that only certain timing exposes;
- leftovers from earlier runs contaminating later ones;
- sequences that break only when tests execute in a particular order.
Deleting the test destroys the witness instead of the problem. Proper procedure mirrors due process: gather evidence (did identical code produce different outcomes?), then either cure the disease or, if the test is beyond saving, retire it deliberately — with a note explaining why, so nobody resurrects it in six months. Between accusation and verdict sits containment, which keeps an unreliable witness off the stand without silencing it forever.
Claim three: “AI has this covered now”
Partially true, and the partial part matters. The strongest documented result belongs to Meta, whose machine-learning selector identified virtually every regression while running roughly a third of the relevant tests — but the same paper notes the model only reached that accuracy after its training data had flaky noise removed beforehand. Discipline in, predictions out. The healing and auto-writing features in vendor demos lack comparable evidence, so treat them as unproven until audited. And even a perfect selector answers only “which tests to run” — never “which tests to delete” or “what blocks a release”. Those judgments stay human, which is why teams switching anything automatic on should log every action it takes. The same pattern — AI accelerating, humans adjudicating — is visible in how AI software development firms are reshaping business innovation.
Claim four: “Hand all the tests to one central team”
The record is unusually blunt here. DORA’s research programme states that developers should own their tests, and explains why: when a central group takes charge of everyone’s tests, pipelines stay broken more often, and programmers quietly drift toward designs that are hard to test at all. Ownership at a distance means defects get blamed rather than prevented. The functioning arrangement looks more like a bar association than a bar: specialists set standards, teach, and take the difficult briefs; the people who write the code answer for its verification, the way an advocate remains responsible for their own case file. Anything else splits knowledge from responsibility, and that split bills itself later, with interest.
Claim five: “Staging is where the truth comes out”
Shared staging is treated like a courtroom where all disputes get settled. The record disagrees on both timing and reliability. Waiting for a staging slot delays the very feedback that prevents defects cheaply, and a shared environment contaminates evidence — one team’s half-finished feature becomes another team’s mysterious failure. Modern practice settles disputes upstream: genuine dependencies exercised in-process, stand-ins for the heavy systems, disposable environments per change, and formal agreements between services verified with contract tooling such as Pact’s can-i-deploy check. Staging keeps one job — a polite final look — and loses its job the day it becomes the gate. Teams hardening their pipelines against risk generally will recognise the theme from this practical DevSecOps learning guide.
Claim six: “Green dashboards prove the suite is healthy”
Counsel presents an immaculate ledger. The cross-examination asks where the failures went. One published account from Slack’s mobile team describes exactly this: suppression of unwanted failures made the numbers beautiful, and a genuine defect walked straight through the filtered gate, shipped broken, and forced a rollback. Records that have been tidied by hand stop being records — they become advocacy. Health is better judged by questions nobody can game: how many red results were false, how long the wait for a trustworthy answer, how old the oldest quarantined test. Those numbers survive scrutiny because nobody curates them.
The TestOps verdict, and what it does not say
On the evidence: the practice is real, the shortcuts are not. Tools assist, they do not adjudicate. Intermittent tests deserve inquiry, not execution. AI earns trust case by case, with logs. Ownership stays with the people who write the code. Staging observes; it does not judge. And an honest record outranks a pretty one, every time. Readers wanting the source material — the term’s origin story, its boundaries and the studies cited above — will find it collected in the Ops Atlas article on TestOps. Judgment reserved, in the best sense: the record now speaks for your team, not about it.
