We Quarantined Flaky Tests — The 6-Month Cost

The flaky test quarantine had 8 tests in it when I joined the project. Two quarters later it had 287. Nobody had ever taken one out.

That is the flaky test quarantine cost nobody puts in the enterprise business case. Not CI minutes — 287 tests out of 1,240 running nowhere, a dashboard reporting a 99.6% pass rate on 77% of the suite, and twelve tests in that pile that had been failing for an extremely good reason. Two of those reasons reached production and came back as customer-service tickets.

A large Canadian retailer: e-commerce plus in-store pickup, 6 SDETs across five product squads, 860 Playwright tests in TypeScript and 380 TestNG API tests in Java, 8 threads, 34 minutes. Nobody on that team made a bad decision. They made the same small, reasonable decision 287 times.

Why does a flaky test quarantine always grow?

Quarantining is cheap and nobody measures it. Un-quarantining is expensive and nobody owns it. Every incentive in a sprint points one way, and unless something actively pushes back, the bucket only gets bigger.

This isn’t a story about one badly run team. Elasticsearch keeps its quarantine list in a public file, muted-tests.yml — a policy-driven, ticket-linked quarantine at one of the more disciplined open-source shops there is. It held 38 muted tests in July 2024. It holds 266 today. Nearly every one carries an issue link. The ratchet still only turns one way.

The incentive nobody priced

Merges were gated on a green pipeline. When a test went red on your PR and it plainly wasn’t your fault, you had two options: pull the trace, reproduce locally, find the race, write the fix — half a day on a flow you don’t own — or add a tag and ship. Four minutes.

The four-minute option isn’t lazy. Locally, it’s correct. Quarantining is the rational move for every individual engineer and a catastrophe in aggregate. On the Java side it was a TestNG group excluded from the suite; on the Playwright side, one flag in the CI script.

<suite name="regression" parallel="methods" thread-count="8">
  <test name="e2e">
    <groups>
      <run>
        <!-- Added Feb 3. "Temporary — remove after sprint 14." -->
        <!-- Still here in August. -->
        <exclude name="quarantine"/>
      </run>
    </groups>
    <packages><package name="com.retailer.e2e.*"/></packages>
  </test>
</suite>

{
  "scripts": {
    "test:ci": "playwright test --grep-invert @quarantine"
  }
}

Two lines of configuration. That’s the entire mechanism by which a quarter of a test suite went dark.

What the dashboard was actually reporting

A skipped test is not a failure. In the JUnit XML that most CI tools parse, it emits a <skipped/> element, not a <failure/> — and most pass-rate widgets divide passes by executions. Quarantined tests were never executed, so the number went up as the quarantine grew.

Tests:      949 passed, 4 failed, 287 skipped
Pass rate:  99.6%
Duration:   34m 12s
Status:     ✅ PASSED

953 of 1,240 is 77%. Nobody had asked the dashboard for that number, so it never showed it. Same failure mode as a green test suite that’s hiding real bugs: the pipeline isn’t wrong, it’s answering a narrower question than the one you think you asked.

How do you audit 287 quarantined tests?

You run them. All of them, many times, and you sort the results into buckets before fixing a single line. Three of us spent two sprints on it — roughly 240 engineer-hours — and the first week went entirely into rebuilding information we had thrown away.

Rebuilding the signal we deleted

A skipped test produces no data. Not a pass, not a fail, not a flake rate — nothing. So before we could triage anything we had to manufacture a signal: run all 287 twenty times against a pinned build and bucket by outcome. Four days of machine time and a weekend of babysitting.

Before you commit to an audit, measure the hole. This takes a minute:

# How many tests are dark? (Playwright / TS)
grep -rE "test\.(skip|fixme)\(|@quarantine" tests/ | wc -l

# And on the JVM side
grep -rE "@Test\(enabled *= *false|@Ignore|@Disabled|\"quarantine\"" \
  src/test/java | wc -l

# The one that starts the conversation — the growth curve
git log --format='%ad' --date=format:'%Y-%m' -S'@quarantine' -- tests/ \
  | sort | uniq -c

That last one is git log’s pickaxe: -S lists every commit that changed the number of occurrences of a string. Piped through uniq -c, it prints how many commits touched the quarantine tag each month — not a precise test count, but the shape of the curve is the point. Ours read 2, 6, 31, 58, 94, 96. Put that on a slide and you don’t have to argue for the cleanup.

The triage rubric

Every quarantined test landed in exactly one of five buckets. The distribution is the whole story:

140 of 287 — nearly half — should never have been quarantined because they should have been deleted. That’s the quiet scandal of a quarantine bucket: most of it is a graveyard nobody had the nerve to dig. Quarantining a dead test feels responsible; deleting it feels destructive. Deleting it is the honest act.

The 74 infra flakes were the usual suspects — the async mistakes that produce intermittent Playwright failures outnumbered every selector problem combined. The 61 environment failures were a payment sandbox and a tax service throwing 504s under parallel load, the sort of thing that disappears once you have a test environment you actually control.

We got that bucket wrong on the first pass. Thirty-odd tests went into “environment” because they only ever failed against the payment sandbox, which is exactly what you’d expect a flaky third party to look like. Two of them were the 3-D Secure bug. We came within one triage meeting of deleting the tests that were reporting it.

Were any of the quarantined tests catching real bugs?

Yes. Twelve of them, mapping to four distinct product defects — and two of those four had already shipped and come back through customer support, three and five weeks after the test that would have caught them was tagged @quarantine.

• Inventory reservation race (5 tests). Two shoppers could both reserve the last unit for in-store pickup. Shipped. Roughly 140 “item unavailable at pickup” contacts to customer service that quarter. The test had been flagging it since week two of quarantine.
• Promo stacking (3 tests). Two percentage-off coupons could combine under a specific cart-edit sequence. Finance found it in reconciliation — about CAD 38,000 in over-discounted orders across four months.
• 3-D Secure redirect (2 tests). One card issuer’s redirect silently dropped about 0.8% of card checkouts. No user ever reported it, because customers who fail at payment don’t file tickets. They leave.
• Wishlist 500s under concurrency (2 tests). Real, intermittent, low blast radius. Fixed in a day once someone looked.

Every one of those tests was doing its job. They failed intermittently, which is what a concurrency bug looks like from the outside — the same lesson as the flaky test that turned out to be a race condition.

This was the predictable outcome. A 2023 study of the Chromium CI (Haben et al.) found flaky tests revealed more than a third of all regression faults in the project — and that flakiness prediction, which condemns the test rather than the individual failure, missed 76.2% of those faults. Condemning the test is precisely what a quarantine tag does.

What is the real cost of flaky test quarantine in an enterprise suite?

The bill has three lines, and only one of them is the line teams budget for. Six months of quarantine cost us 287 dark tests — 23% of the suite, reported to leadership as a 99.6% pass rate — 240 engineer-hours to dig back out, and four production defects, two of which reached customers before anyone looked. What it bought in return was about 60 hours of CI runner time.

Skipping 287 tests held the pipeline at 34 minutes instead of roughly 42. Whatever you think an SDET-hour costs, eight minutes is a terrible price to pay in coverage — and there are five better reasons a suite is slow, none of which require going dark.

What quarantine policy actually stops the bucket from growing?

Four rules: quarantined tests still run; no ticket and no cap means no quarantine; two sprints, then fix or delete; and the count sits next to the pass rate. The first is the only one that’s really an idea — the rest are enforcement.

Rule 1: Quarantined tests still run

Quarantine is not skip. A quarantined test executes on every run, reports its result, and simply doesn’t gate the merge. That distinction is first-class in the tools that take it seriously: Datadog’s flaky test management separates quarantined (runs, doesn’t fail CI) from disabled (doesn’t run), and Buildkite Test Engine separates muted from skipped. TestNG and JUnit give you one switch — off — and nothing in between.

We thought we were quarantining. We were skipping and calling it quarantine. A skipped test gives you nothing; a running-but-non-blocking test gives you a flake rate, which is the number that separates infrastructure from a bug. It would have told us about the inventory race in week two instead of month five.

Playwright won’t do this for you — the feature request for exactly this behavior was closed without being implemented, and people are still asking. You wire it yourself with a second project:

export default defineConfig({
  projects: [
    {
      name: 'blocking',
      grepInvert: /@quarantine/,
    },
    {
      // Still executes. Still reports a flake rate.
      name: 'quarantine',
      grep: /@quarantine/,
      retries: 0,
    },
  ],
});

The non-blocking half has to happen in CI, not in the config — Playwright runs every project in one process and exits non-zero if anything anywhere fails, so there’s no per-project soft-fail to configure. Two separate invocations, and only the first one gates:

{
  "scripts": {
    "test:ci": "playwright test --project=blocking",
    "test:quarantine": "playwright test --project=quarantine || true"
  }
}

On the TestNG side it’s the mirror image of the exclusion that started all this — include instead of exclude, in its own non-blocking nightly suite that feeds the flake-rate dashboard.

Rule 2: No ticket, no quarantine. No cap, no quarantine.

The tag carries a ticket key, and CI refuses the build if it doesn’t. Since Playwright 1.42 the tag is a first-class test option rather than a string stuffed into the title, and the ticket goes in an annotation:

test('rejects two stacked percentage-off coupons', {
  tag: '@quarantine:RTL-1482',
  annotation: { type: 'issue', description: 'https://jira/RTL-1482' },
}, async ({ page }) => {
  await checkout.applyCoupon('SAVE20');
  await checkout.applyCoupon('EXTRA10');
  await expect(checkout.total).toHaveText('$79.99');
});

The same check enforces a hard cap — 12 tests, about 1% of the suite. Want to quarantine the thirteenth? Retire one of the twelve first.

#!/usr/bin/env bash
set -euo pipefail

CAP=12
TAGS=$(grep -rhoE "@quarantine(:[A-Z]+-[0-9]+)?" tests/ src/test/java || true)

# printf, not echo — `echo ""` emits a blank line that grep -vc counts as untagged
COUNT=$(printf '%s' "$TAGS" | grep -c . || true)
UNTICKETED=$(printf '%s' "$TAGS" | grep -vcE ":[A-Z]+-[0-9]+$" || true)

[ "$UNTICKETED" -eq 0 ] \
  || { echo "::error::$UNTICKETED quarantined test(s) with no ticket key"; exit 1; }

[ "$COUNT" -le "$CAP" ] \
  || { echo "::error::quarantine is $COUNT, cap is $CAP — retire one first"; exit 1; }

echo "quarantine: $COUNT/$CAP, all ticketed"

The cap is the load-bearing rule. An SLA without a cap is just a queue; a cap forces a trade every single time.

Rule 3: Two sprints, then fix or delete

Every quarantine ticket gets a due date four weeks out — two, if you’re still digging out of a backlog — and exactly two outcomes at expiry: fixed, or deleted. There is no “extend.” A test nobody will fix in a month is a test nobody believes in, and the delete option is the one that makes the policy work: deleting is visible and arguable, while skipping is invisible forever.

Rule 4: Put the quarantine count next to the pass rate

Rule 1 is necessary and not sufficient. Google, Microsoft and Datadog all say “quarantined tests keep running, so you keep the signal” — true right up until you notice that a result nobody reads is not a signal, it’s a log line. Running the tests keeps the data; watching the number keeps the team.

So executed coverage sits on the same dashboard row as pass rate, with a Slack alert when the count crosses 8. Watch the derivative, not the level — the industry’s “keep quarantine under 5%” rule of thumb is repeated everywhere and traced to nowhere. Visibility is what actually changed behavior: within a month, engineers were opening PRs deleting quarantined tests with nobody asking them to, because the count was in front of them every morning.

Results

Six months on, the suite is 1,010 tests instead of 1,240 — 140 deleted, 90 consolidated. Smaller and honest beats bigger and dark. The pipeline runs in 26 minutes instead of 34, because the 61 third-party tests moved to a nightly contract run. The dashboard now reports executed coverage, and it reads 100%: the quarantine lane runs, it just doesn’t gate. Median time in quarantine is 6 days. Nothing has hit the four-week expiry yet.

What I’d Do Differently

Ship the alert before the audit. I ran the cleanup first, then spent two weeks re-justifying it to a director who reasonably asked why 240 hours of SDET time was going into tests that “weren’t even running.” Had the count been on the dashboard from day one, the graph would have made that argument for me. Instrument first, then fix.

Never let a quarantined test stop executing. Reconstructing six months of deleted failure data — 287 tests × 20 runs on a pinned build — cost more than keeping those tests running non-blocking would have cost for the entire six months. We paid to rebuild information we’d been given for free and thrown away.

Delete harder, earlier. I argued to keep some of the 22 “nobody knows what this asserts” tests, on the theory that someone might reconstruct the intent. Nobody ever did. They were deleted four weeks later anyway, after burning three triage meetings.

Go Count Your Skips

The uncomfortable part isn’t the 287. It’s that not one person on a good team ever ran the number. It wasn’t hidden. Nobody looked.

Your one action today: run the git log -S pickaxe command from the audit section against your own test directory and print the per-month count. If the line only goes up, you don’t have a quarantine. You have a skip list wearing a lab coat, and somewhere in it is a test still trying to tell you something.