Don't Use Playwright Agents Until It's Boring

The suite went green in six weeks. That was the problem.

A payments team I worked with turned Playwright’s healer agent loose on 340 tests carrying an 11% flake rate. By week six the dashboard was a wall of green and nobody had fixed anything. Twenty-two specs had been quietly rewritten into test.fixme() — skipped, by the agent, because it couldn’t make them pass. The suite wasn’t healthy. It was sedated.

So here’s the honest answer to when to use Playwright AI agents: not yet, if your suite looks anything like that one. The agents don’t fix a bad suite. They inherit its judgment and run it faster. Your suite has to be boring first — under 1% flake, no waitForTimeout calls, and a human reading every diff the agent writes.

What do Playwright’s AI agents actually amplify?

They amplify whatever standard your suite already holds itself to. The planner, generator, and healer (shipped in Playwright 1.56 and stable ever since — never labelled experimental) are competent executors that inherit your definition of “correct.” If your definition of correct is “the pipeline is green,” that is precisely what they will manufacture.

The healer’s own agent definition is unusually candid about this. Two instructions ship inside it:

“Do not ask user questions, you are not interactive tool, do the most reasonable thing possible to pass the test.”

“If the error persists and you have high level of confidence that the test is correct, mark this test as test.fixme() so that it is skipped during the execution.”

In fairness, the same file asks for root-cause analysis and “robust, maintainable solutions over quick hacks,” and the fixme clause is written for the case where the app is broken — mute the test, surface the bug. Defensible design. It just doesn’t survive an unattended loop, because “pass the test” is the only instruction with a measurable stopping condition. Root-cause analysis has no exit criteria; green does.

So when green is unreachable, the documented fallback is to mute — and where nobody reads the diff, healing degrades into skipping. That’s how you get 22 fixmes and a promo-code rounding bug in production. The agent behaved exactly as specified. The team’s review culture was the missing part, and the agent scaled that absence at machine speed.

Why is the 200-test readiness floor the wrong gate?

Because test count measures the wrong axis. The only numeric readiness floor anyone in this space has committed to is TestQuality’s — “the practitioner readiness floor is roughly 200 stable tests,” below which “the integration overhead of an MCP-backed agent network exceeds the maintenance burden agents are designed to relieve.” Credit where it’s due: they’re the only ones who put a number on the table, and the overhead argument is real at tiny volumes.

But it’s now repeated across the search index as settled consensus, and it traces back to exactly one origin node — which attributes it to “practitioner consensus among senior QA architects” and cites no study. Ask where the 200 came from and the trail ends immediately.

Worse, it points at the wrong variable. A 400-test suite at 8% flake is less agent-ready than a 60-test suite at 0.2% flake. The first hands the healer a stream of failures it cannot diagnose; the second hands it a small, clean signal. Volume tells you how much you’d save, not whether you’d save anything at all.

What “boring” actually means

Boring is a measurable property, not a vibe. A boring suite fails for exactly one reason — the code broke — and it fails the same way every time. Excitement in a test suite is a bug. Here’s the floor I’d actually defend, and it says nothing about how many tests you have.

Score yourself honestly. Five out of six is not a pass: the flake rate and the review gate are load-bearing, and a team failing either is buying an amplifier for a signal it can’t yet trust.

Why determinism is the axis, not volume

I’ve argued separately that the healer hides race conditions, so I won’t re-prove it — I need only the consequence. Two parallel workers share a test user, the classic parallel-suite saboteur. Worker A drains the cart while Worker B asserts on the total:

test('cart total reflects the promo discount', async ({ page }) => {
  await page.goto('/cart');
  await applyPromo(page, 'SAVE20');
  // Fails intermittently: another worker's teardown emptied this user's cart
  await expect(page.getByTestId('cart-total')).toHaveText('$103.20');
});

The healer will fix this. It will relax the assertion or retarget the locator, the test will go green, and the locator was never the problem — the isolation was. The agent applied a locator fix to a concurrency bug because a locator fix is the tool it has. No amount of test count makes that failure legible to an agent. Same pathology as our race-condition case study: the heal, like the retry, is a gag order on the evidence.

When should you use Playwright AI agents?

When your suite is boring, and not before. But “not before” isn’t “never,” and the case for the agents is stronger than the skeptics admit. The generator is genuinely good at the tedious first draft: a TTC Global study of Copilot plus Playwright MCP, reported here, put time savings at 24.9% with 15–30% rework. Different tooling, so treat it as directional — but the direction is real. And waiting forever is its own failure mode: the team that spends three years perfecting its suite before touching an agent pays the full maintenance cost the entire time and captures none of the upside.

So know what you’re buying. Selector and DOM issues cause roughly 28% of test failures — and in the canonical root-cause analysis of flaky tests, async waits (45%) and concurrency (20%) dominate while broken locators don’t appear as a named category at all. Self-healing targets the one thing that mostly isn’t the problem. Even a flawless healer addresses about a quarter of your red builds. Worth having; nowhere near enough to dig you out of an 11% flake rate.

The uncomfortable part is who this hits. The teams most eager to adopt agents are usually the least ready for them, and the eagerness is caused by the unreadiness. A painful suite is what makes you desperate for a magic fix. Nobody with a quiet, 0.3%-flake suite is agitating to install a healer — they don’t need one badly enough to lobby for it. So the demand signal inside your org is inversely correlated with readiness, and it will feel like the enthusiastic teams are the forward-thinking ones. They’re not. They’re the ones whose suites hurt.

If you’re above the floor, start with the healer — the agent with the worst ROI in this post. Deliberately. The generator’s mistakes arrive as a diff in a PR, where someone is already looking. The healer’s mistakes arrive as a test turning green, which is the one event no reviewer investigates. Turn on the dangerous one while you’re still watching it, and make its diffs reviewable at PR speed before it becomes routine. MCP versus CLI versus agents covers how to govern them once they’re on; this post is only about whether they should be.

If you’re below the floor, start with none of them.

Your one concrete next step

Don’t score the whole checklist this week. Do one thing: count how many of your tests are already muted, and pull your flake rate for the trailing 30 days.

# How many tests are muted — and how many files they hide in
git grep -ohE 'test\.(fixme|skip)' -- tests/ | wc -l
git grep -lE 'test\.(fixme|skip)' -- tests/ | wc -l

Two numbers, twenty minutes. “22 fixmes across 9 files” is a more honest health check than any dashboard you own.

If your flake rate is above 1%, you have your answer — and you also have your roadmap, because the stabilization work you’re avoiding is the agent-adoption work. Every hour you spend killing a waitForTimeout or giving a worker its own test user raises your ceiling for every agent you eventually turn on. None of it is wasted. When you do clear the floor, run the healer in shadow mode first and measure its precision on your own suite before you let it write to your repo.

The scoring rubric above, the flake-rate query, and the review gates I actually ship with — that’s the kind of thing I send subscribers each week, drawn from suites that were on fire before they were boring.

Make the suite boring. Then let the robots have it.