Run Playwright's Healer in Shadow Mode Before You Trust It

Go looking for Playwright’s Healer heal rate and you’ll find one — somewhere around 75%, repeated across a dozen blog posts. Now try to find where it came from. Microsoft’s Test Agents documentation publishes no success rate at all. The number is a citation loop: posts sourcing posts, with no primary source at the bottom. So the headline figure is unsourced — and the figure that should actually keep you up at night was never published by anyone: how often the agent is confidently wrong, patching a red test to green by binding to a similar-but-functionally-different element. You need your own precision number. Nobody can hand you one.

Here’s the move that nobody in the Playwright AI tooling threads is making: borrow shadow deployment from MLOps. You wouldn’t promote a new fraud-detection model to production on the vendor’s accuracy claim — you’d run it in shadow against live traffic, log what it would have done, and measure precision before it makes a single real decision. The Healer agent is a model making promotion decisions about your test coverage. Treat it like one.

Why is a published heal rate the wrong number to trust?

Two reasons, and the first one is embarrassing: the heal rate everyone quotes isn’t sourced. Microsoft’s Test Agents documentation describes what the healer does and publishes no percentage. Trace the ~75% figure back and it dissolves into blogs citing blogs.

The second reason survives even if you accept the number anyway. A heal rate is an aggregate, and your coverage risk lives in the tail. The agent’s own role definition on GitHub is candid that its job is to make tests pass, not to verify intent — so a heal rate counts tests turned green, which is precisely the metric that cannot distinguish a repair from a cover-up. The misses aren’t random either: they cluster in the suites where you can least afford a silent failure — legacy admin panels, iframe-heavy flows, anything with duplicated text in the DOM.

The failure mode is specific and ugly. A locator breaks because the dev renamed data-testid="submit-quote" to data-testid="submit-policy". Healer scans the page, finds a button reading “Submit,” binds to it, and the test goes green. Except that button was in a comparison modal, not the quote form. Your test now asserts a happy path it never exercises. This is the same disease as green tests that hide real bugs, except now an agent is manufacturing the green at scale, overnight, in CI.

The Playwright team is explicit that Healer output should go through a PR with human review — Debbie O’Brien’s walkthrough of the Healer agent makes this point directly. But “review every PR” doesn’t scale when the agent generates 40 patches a night, and human reviewers rubber-stamp green diffs after week two. You need a precision number to decide which suites earn automation and which stay manual. That number is what shadow mode produces.

What does shadow mode mean for a self-healing agent?

It means running Healer in observe-only: let it propose a fix for every red test, record the proposal, but never apply it to the suite that gates your pipeline. In ML, shadow deployment duplicates production traffic to a candidate model whose outputs go to a log instead of to users — the candidate makes predictions, you grade them, nobody gets hurt if it’s wrong. Deepchecks’ definition of shadow deployment frames it as the first safe look at real-world behavior before promotion.

Map that onto testing one-to-one. The “production model” is your human SDET, who fixes the broken locator the correct way. The “candidate model” is Healer, proposing a fix in parallel. Both react to the same input — a red test. The human’s fix ships. Healer’s fix goes to a ledger. Later you compare them and compute precision: of the patches Healer proposed, how many matched the element the human actually targeted?

The mechanical catch: Playwright ships no --shadow flag. There’s no observe-only mode built in. So you build it the way enterprise SDETs build everything — from the diff. Run Healer on a throwaway branch, capture the locator changes it makes, then git reset --hard so nothing persists. The proposal is the diff. That’s the whole trick, and it’s why this works without any cooperation from the agent itself.

How do you build the shadow-mode harness?

Four moving parts: a throwaway-branch runner, a diff parser, a labeling step, and a precision rollup. None of it is exotic. The discipline is in running it for weeks before you draw conclusions, not in the code.

The shadow runner is a thin wrapper. It exists only to make Healer commit its decision somewhere you can inspect it without consequences:

#!/usr/bin/env bash
set -euo pipefail

# Run on a detached branch so nothing the agent writes can reach main.
SHADOW="healer-shadow-$(date +%s)"
git switch -c "$SHADOW"
npx playwright heal --suite "$1" || true     # let it patch the throwaway tree
git diff --unified=0 -- 'tests/**' > "shadow/${1}-$(date +%s).diff"
git switch - && git branch -D "$SHADOW"       # destroy the branch, keep the diff

That git diff of the throwaway branch is the proposal. Parsing it into a ledger entry is where the judgment lives — you want the old locator, the new one, and enough provenance to compare models later when your org bumps from one Claude or GPT version to the next:

interface HealerProposal {
  testFile: string;        // tests/quote/submit.spec.ts
  failingLocator: string;  // getByTestId('submit-quote')
  proposedLocator: string; // getByRole('button', { name: 'Submit' })
  model: string;           // claude-sonnet-4-6 — track this across upgrades
  proposedAt: string;      // ISO timestamp
  // Filled in later, by a human, once the real fix lands:
  humanLocator?: string;
  // 'match' | 'mismatch' | 'na' — the only column that matters
  verdict?: "match" | "mismatch" | "na";
}

Why the rollup refuses to average

The precision rollup is deliberately boring. Group by suite, divide matches by gradeable proposals, and refuse to average across suites — because the average is the lie that gets you into trouble:

function precisionBySuite(ledger: HealerProposal[]) {
  const bySuite = groupBy(ledger, p => suiteOf(p.testFile));
  return mapValues(bySuite, rows => {
    const graded = rows.filter(r => r.verdict !== "na");
    const matched = graded.filter(r => r.verdict === "match");
    // Precision = correct fixes / fixes the agent was confident enough to make
    return { n: graded.length, precision: matched.length / graded.length };
  });
}

What does the precision data actually tell you?

That a single org-wide trust policy is malpractice. When I ran this for six weeks on a Canadian insurer’s policy-administration platform, the blended precision came out close to the textbook figure — and it was useless. The split underneath it was the entire story.

The design-system suite — modern components, disciplined data-testid usage — healed at 88% precision. The legacy admin suite, all nested iframes and three buttons reading “Save” on one screen, healed at 43%. Any blended figure averages those two into a number that describes neither, and a “Healer is reliable enough, let it auto-PR everywhere” policy would have shipped silent coverage loss into the exact suite covering the highest-risk workflows: policy edits an underwriter relies on. Those testingtools.ai folks make the same argument from the metrics side — their five metrics for self-healing performance put precision and false-positive rate ahead of raw heal count, for exactly this reason.

The precision number isn’t there to grade Healer. It’s there to grade your test design. A suite that heals at 43% is telling you its locators are tied to rendered structure instead of stable contracts. The fix isn’t a better agent — it’s data-testid attributes and fewer duplicate labels, which is the same groundwork that makes self-healing locators work at all. Shadow mode turns a vague “our admin tests are flaky” into a defensible “this suite is not eligible for automated healing until precision clears 80%.”

How do you graduate a suite from shadow to auto-merge?

Tiered trust, gated on the suite’s own measured precision — never a global toggle. Promotion mirrors canary rollout for ML models: you don’t flip from shadow straight to fully automatic, you step through increasing levels of autonomy and watch for regressions at each one. Qwak’s breakdown of shadow vs. canary is the mental model — shadow validates, canary controls blast radius, and you move between them on evidence.

Two rules that keep the table honest

Two rules make the table safe rather than decorative. First, precision is a rolling window — a suite that drops below its tier for two consecutive weeks demotes automatically, no meeting required. Second, the top tier never goes fully hands-off; even at 90%+, a healed test enters a canary lane and only counts as real coverage after a mutation check confirms it still fails when the underlying behavior breaks. Green is not coverage. A test that can’t fail isn’t testing anything.

What I’d do differently

I ran shadow mode globally for the first two weeks and drowned in noise — proposals from suites I was never going to automate, doubling CI cost across the board for data I didn’t need. Scope it from day one: pick the two or three suites where automated healing would actually save real SDET hours, shadow only those, and leave everything else on manual. The goal is a precision number for the suites that matter, not a science fair.

I’d also wire the labeling step into the existing PR flow instead of treating it as a separate chore. When an SDET fixes a broken locator, a bot drops a one-line prompt on the PR: “Healer proposed getByRole('button', { name: 'Submit' }) — match or mismatch?” One click. Labeling that rides along with work people already do gets done. Labeling that lives in a separate spreadsheet does not.

The deeper lesson: self-healing isn’t a feature you turn on, it’s a capability you earn per suite by proving your locators are stable enough to heal correctly. The agent is only as trustworthy as the test design beneath it, and shadow mode is how you find out — with a number — before that gap ships to production.

Your one concrete next step

Pick the single suite where you’re most tempted to turn on auto-healing. This week, run Healer against its next batch of real failures on a throwaway branch — git switch -c shadow, heal, capture the diff, git branch -D — and hand-label ten proposals as match or mismatch against how you’d have fixed them. Ten data points from your own suite will tell you more than any vendor’s 75%. If even three of those ten are mismatches, you just found out auto-merge would have shipped a 30% silent-failure rate into that suite — and you found out in an afternoon instead of in an incident.

If you want the longer build — the full ledger schema, the canary lane, and the mutation guard wired into GitHub Actions — that’s the kind of thing I send subscribers. Get battle-tested QA automation patterns like this one weekly: no fluff, just what’s actually worked across enterprise suites.