We Budget 30% of Every Sprint for Flaky Tests

The director didn’t say no to test maintenance. He said, “The tests are already written.” Then he approved the coverage roadmap and left the room, and I spent the next two quarters paying for a flaky test maintenance budget that existed in every way except on paper.

That is what a flaky test maintenance budget looks like when it doesn’t exist. You didn’t lose that line. You never got to argue about it, because the work is real, it happens anyway, and it comes out of the only capacity you were funded for. We now reserve 30% of every sprint for it, explicitly, on the board, in front of the product owner. Our new-test output didn’t drop. What changed is that we stopped looking slow.

Why does flaky test maintenance always lose the sprint planning argument?

Because it isn’t in the argument. Nobody rejects the maintenance ask on the merits — they reject it against an alternative, and unbudgeted work has no advocate in the room. Coverage has a roadmap, a stakeholder, and a slide. Fixing a locator that broke when marketing shipped a new nav bar has none of those, so it loses to a comparison it never entered.

Unbudgeted work doesn’t look like work — it looks like slowness

This is the part that ends QA careers. At a large North American freight company — 5 SDETs, 4 product squads, 1,140 tests across Playwright and TestNG, 36-minute pipeline — we spent an entire quarter doing flaky-test work we hadn’t asked permission for. The tests got fixed. The sprint commitments didn’t get hit: 61% of committed points landed, quarter over quarter.

Nobody read that as “they are absorbing unfunded work.” They read it as a throughput problem, and two of my SDETs got throughput language in their performance reviews for time they spent keeping the suite honest. Work you do without a budget line is not invisible — it is visible as a defect in you.

How much sprint capacity should a flaky test maintenance budget get?

Start at 30% of QA capacity, then earn the right to lower it. That number isn’t a benchmark I read — it’s what we measured when we finally logged the work, and it sits deliberately above the 15-20% that most published guidance recommends and below the 30-50% that teams actually report burning. The gap between those two ranges is the whole problem: the recommendation describes a healthy suite, and you don’t have one yet.

Do the arithmetic before you ask for the money

You can model most of this from two inputs: how many red builds per sprint turn out to be flake, and how long a human spends deciding that. Slack published the second number — 28 minutes per manual triage across their mobile CI. Here’s the same math on our suite, for one two-week sprint:

The formula says 21%. Our six-week log said 31%. That gap is the context switching the model can’t see — the third time in a day you stop writing a new test to explain that no, the checkout suite isn’t broken, it’s the shared staging environment again. Ask for 30%, and defend it with your log, not my table.

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  // 1.52+: a test that only passed on the retry now fails the run.
  // You cannot budget for a cost your config is hiding from you.
  failOnFlakyTests: !!process.env.CI,
});

”We’re spending 30% of QA on tests that already exist?”

Yes — and a leader who finds that alarming is right. A 30% flake load is not a thing to fund forever; it’s a symptom, and the honest response is partly to kill the cause rather than permanently subsidize the treatment. Any QA lead who wins this argument and then defends the 30% line for three years has stopped being useful.

The 30-40% figure that gets quoted as an industry constant is not physics, either. Google runs at roughly 1.5% flaky test runs, and I’ve seen a 1,500-test suite held under 1% by a team that simply owned its own data-testid attributes. The 30% is a bill for decisions made upstream of you — locators you don’t control, environments you don’t own, test data nobody maintains. Fund it, and then go spend it on the upstream.

The exit condition that makes it a budget instead of a subsidy

Ours: when the flake rate stays under 2% of executed tests for two consecutive sprints, the carve-out drops by 10 points at the next planning cycle. That single sentence is what turned a “no” into a “fine, show me in a quarter” — because it made the 30% a claim I could be held to, rather than a permanent ask.

That last number is the honest one, and it’s the one I’d lead with if I ran the meeting again. New-test output went up by two. It couldn’t have gone much further either way — we had been spending that 30% all along, just without permission. The budget didn’t cost the org new coverage. It made the coverage forecast true for the first time.

The uncomfortable finding came out of the triage log: roughly 40% of what we’d been calling “flaky” was not a test defect at all. It was product race conditions and environment drift. That’s the argument’s real payoff — a test that fails intermittently and passes on retry is usually telling you something, and an unfunded team never has the hours to listen. They suppress instead, which is how you end up with a green suite that’s hiding real bugs.

What does the budget buy that a quarantine policy alone can’t?

Enforcement. Every quarantine policy has a fix-or-delete SLA in it, and every one of them fails the same way — not because the rule was wrong, but because nobody was given the hours to satisfy it. An SLA is a promise about capacity. If the capacity isn’t on the board, the SLA is decoration.

I’ve watched that happen, and written up what six months of an unfunded quarantine policy actually cost. The short version: a skip list is the only thing in a test suite that grows without anyone deciding to grow it. That team wrote the policy and skipped the budget, which is the same thing as writing no policy at all. This is also the structural answer to the accountability gap shift-left left behind — QA absorbing maintenance for breakage it didn’t cause, and calling it a work ethic.

Start with the log, not the ask

Don’t walk into next planning and ask for 30%. You’ll lose, the same way I did, to the sentence “the tests are already written” — because right now you have an opinion and they have a roadmap.

Go label six weeks of maintenance hours instead. Then walk in with your own number, whatever it turns out to be, and the exit condition attached to it. The argument stops being about whether maintenance deserves capacity and becomes about whether the invoice you’re holding is accurate. That’s an argument you win, because you’re the only person in the room who did the measurement.