The software industry has reached a quiet consensus on flaky tests: find them, fix them and move on.
I believe that consensus is fundamentally wrong.
I’ve spent years building and operating test execution infrastructure at a major financial institution, supporting over 1M automated tests monthly across 25,000+ engineers. In high-scale, regulated environments, unexamined signal failures aren’t just a developer annoyance. They represent unquantified release risk. We keep treating flakiness as a testing problem. It isn’t. It’s a trust problem. And until engineering leaders make that distinction, no amount of test fixing will contain it.
What the Industry Gets Wrong
The standard playbook is familiar to anyone who’s managed quality at scale. A test starts failing intermittently. Someone files a ticket. The test gets quarantined, retried, or deleted. The pipeline turns green. Everyone moves on.
That approach mistakes the symptom for the disease.
Flaky tests don’t erode engineering velocity by failing. They erode it by making engineers stop trusting their own automation.I’ve watched this happen, not as a theoretical risk, but as a cultural shift that takes months to reverse once it sets in.
When engineers learn that red doesn’t reliably mean failure , they no longer treat the pipeline as a signal. They rerun before investigating. They merge on instinct rather than evidence. They build informal mental models of tests they don’t trust that never get documented anywhere. The pipeline shows green. The green is meaningless. And no one files a ticket because the behavior has become the norm.
At around 1 million tests per month, a 1% failure rate translates into close to 10,000 bad signals. The suppression behavior that develops around those signals doesn’t stay contained. It spreads across teams, pipelines and eventually the entire organization’s relationship with automated testing.
The Retry Trap
The industry’s favorite solution makes this worse, not better.
Automatic retries are standard in virtually every CI/CD platform now. The logic seems reasonable: retry intermittent failures and only surface persistent ones. I used to think this made sense too.
It doesn’t. Retries don’t solve flakiness. They hide it.
Every retry that converts a flaky failure into a passing result removes a data point that could have led to root cause identification. Over time, retry policies create a false floor. A pipeline that looks stable because its instability is being systematically suppressed.
Consider the following scenario that I have witnessed more than once. The infrastructure timing problem intermittently caused one of the critical tests of an integration to fail. The CI system retries the test and it passes.No investigation opens because the dashboard is green. A week later the same pattern appears in another service, then another. Each incident gets dismissed as just another flaky test because the retry mechanism is masking the underlying signal. What looked like isolated instability eventually traces back to a shared infrastructure bottleneck affecting dozens of pipelines. The engineering organization didn’t lack data. It lacked visibility into the data that retries had systematically hidden.
I’ve watched organizations invest heavily in retry infrastructure while their actual flakiness rates kept climbing. The pipeline looked healthier. The engineering behavior kept getting worse. The retry policy had been optimized for green pipelines, not trustworthy ones.Those aren’t the same thing.And confusing them is at the heart of why flakiness persists at scale.
The Leadership Failure Nobody Wants to Name
Here’s the opinion I expect to be unpopular: the persistence of flakiness at enterprise scale is primarily a leadership failure, not an engineering failure.
Flakiness that isn’t measured doesn’t get managed. In most organizations I have worked with, it is simply not measured. Not because engineers do not know that it is a problem, but because engineering leaders haven’t demanded the metrics needed to make it visible.
Pass rate is easy to measure. Retry rate, false failure rate, mean time to root cause, signal reliability by team and suite. These require instrumentation investment that doesn’t happen without leadership prioritization. And leadership doesn’t prioritize what it can’t see on a dashboard.
The result is a hidden tax that compounds silently. Engineering hours disappear into investigation loops. Pipeline capacity gets consumed by reruns. Deployment velocity erodes through the accumulated friction of signals nobody trusts. In a regulated environment like financial services, add the compliance risk of defects escaping through suppressed signals. None of this appears in a sprint report. All of it is real.
What Needs to Change
In my experience, three shifts matter more than any specific tooling decision.
Stop measuring pass rate and start measuring signal reliability. A pipeline with a 98% pass rate and a 15% retry rate isn’t a healthy pipeline. It’s a pipeline with a hidden reliability problem that leadership has decided not to look at. Retry rate is the metric that reveals what pass rate conceals.
Stop routing flakiness to test authors and start routing it to root cause owners. Timing failures, data dependency failures, environment failures and infrastructure contention all have different owners and different fix paths. Treating them identically ensures none of them get fixed efficiently.
Stop celebrating green pipelines and start demanding trustworthy ones. Green is easy. Trustworthy is hard. The organizations that understand the difference are the ones that actually ship software with confidence.
Conclusion
The industry has spent years optimizing for pipelines that look healthy. It’s time to optimize for pipelines that are healthy.
Flaky tests aren’t a testing problem to be fixed test by test. They’re a trust problem to be solved at platform level, with instrumentation, classification and signal integrity metrics that give engineering leaders the visibility they need to act.
At over 1M executions a month, intuition isn’t a quality strategy.
