Let AI Triage Your Playwright Failures, Not Decide Them

A 400-spec Playwright suite goes red on the 9 a.m. CI run. Forty-one failures. Some are a renamed data-testid, some are a slow staging deploy, and one — exactly one — is a real regression in the checkout total. The job that eats your morning isn’t fixing that bug. It’s finding it. An LLM can shrink that search from an hour to minutes, and that’s the highest-leverage place to put AI in a test pipeline today. But the instant you let the model emit a verdict instead of a summary, you’ve rebuilt the same bug-hiding machine I’ve spent a decade tearing out of enterprise suites.

That’s the whole argument of this post: use the model to triage — cluster, summarize, route — and keep the decision of flaky-vs-real with a human or a deterministic rule. The two look similar in a demo. At scale they are opposites.

Why is failure triage the real bottleneck, not test execution?

Because execution is already cheap and triage never got cheaper. The wall-clock cost of a wide CI suite has been falling for years, but the human cost of interpreting a red run has not — a person still has to open each failure, read the trace, and decide whether it matters. That decision is the bottleneck, and it scales linearly with failure count while your headcount doesn’t.

The numbers are worse than most leads admit out loud. The proportion of teams reporting test flakiness grew from 10% in 2022 to 26% in 2025 per Bitrise’s analysis of over 10 million builds. Flaky failures cost an average 100-person engineering team roughly $2.6M a year in lost productivity, and QA engineers spend 20–30% of their work week just triaging failures. The median triage for a single flaky failure runs about 28 minutes — read logs, check if it’s known-flaky, decide retry or investigate.

Multiply 28 minutes by 41 failures and you’ve lost the morning before you touch the one bug that shipped. This is the task an LLM is genuinely good at: reading a pile of unstructured artifacts fast and grouping them. It’s also the task where letting it go one step too far is catastrophic.

What should an LLM triage layer actually do?

It should turn a wall of failures into a ranked, clustered, human-readable brief — and stop there. Concretely: dedupe failures that share a root cause, attach the relevant evidence, draft a one-line hypothesis per cluster, and order them by “most likely to be a real defect.” Nothing in that list is a verdict. Every item still lands in front of a person or a deterministic gate.

The good news is you don’t need a new platform for this. Playwright’s custom reporter API already hands you everything the model needs inside onTestEnd, including the path to each test’s trace and attachments.

import type { Reporter, TestCase, TestResult } from '@playwright/test/reporter';

class TriageReporter implements Reporter {
  private failures: TriagePayload[] = [];

  onTestEnd(test: TestCase, result: TestResult) {
    if (result.status === 'passed' || result.status === 'skipped') return;
    const trace = result.attachments.find((a) => a.name === 'trace');
    this.failures.push(this.toPayload(test, result, trace?.path));
  }
}

The result.attachments array is the part people miss. It carries the trace, screenshots, and video paths — the exact artifacts a human opens during triage. You extract the cheap, high-signal slices (error message, failing step, last console errors) into a compact payload, and keep the heavy trace as a link the model can ask a human to open, not something you stream into the prompt.

Build the payload from signal, not from everything

The mistake here mirrors the one I covered in Playwright MCP vs CLI vs Agents: letting a full page snapshot into context, where it rides along on every later turn. Current Playwright MCP writes action snapshots to a file and only returns the full tree inline when the agent calls browser_snapshot, but one oversized snapshot is still enough. Triage has the same trap. You don’t need the full trace in the prompt — you need the error, the failing action, and a few lines of console.

private toPayload(test: TestCase, result: TestResult, tracePath?: string) {
  return {
    title: test.titlePath().join(' › '),
    file: test.location.file,
    error: result.error?.message?.slice(0, 800) ?? '',
    // Console output is where race conditions and 500s confess
    console: result.attachments
      .filter((a) => a.name === 'console')
      .map((a) => a.body?.toString().slice(0, 400)),
    tracePath, // a link for a human — NOT prompt fuel
    retries: result.retry,
  };
}

Notice retries. Whether a test passed on retry is a deterministic, free signal — you don’t need an LLM to know a test was flaky-on-retry, and you shouldn’t pay one to guess it. Feed facts you already have as facts; reserve the model for the messy clustering and summarizing.

Constrain the model to a brief, not a judgment

The prompt design is where the discipline lives. Ask for structured triage output with an explicit needs_human escape hatch, and forbid a binary flaky/real label entirely.

You are triaging Playwright CI failures. For each cluster, output:
- cluster_id and the failing tests in it
- one-line root-cause HYPOTHESIS (not a conclusion)
- evidence: which error/console lines support it
- signal_strength: how much the evidence supports the hypothesis
Do NOT label a failure "flaky" or "real bug".
Do NOT recommend retrying or closing anything.
If evidence is thin, set needs_human: true.

The model returns a brief. A human (or a rule) decides. That single constraint is the difference between a tool that saves your morning and one that quietly deletes your regressions.

Where is the line you must not cross?

The line is the verdict. Triage means “here is what I think this is and why” — a decision means “this is flaky, I retried it, the build is green.” The moment the model owns the verdict, its success signal becomes the build went green, not the code is correct. That’s the identical failure mode I wrote about in Playwright’s Healer agent hiding race conditions: a loop optimizing for green cannot distinguish a locator drift from a real defect, because both end the same way when you stop looking.

The research here should end the debate. A study of flaky-test prediction on the Chromium CI found methods hitting 99.2% precision and 98.4% recall at detecting flaky tests — numbers that would sail through any model review. Yet the same methods labeled roughly 76% of genuinely fault-triggering failures as flaky. The classifier was excellent at its stated job and catastrophic at the one that matters, because “looks like noise” and “is a real bug behaving like noise” are the same shape from the outside. I’ve lived this: the failures that hurt most in a parallel suite — covered in our breakdown of shared test users in parallel Playwright — are precisely the ones that look flaky and aren’t.

This is also why “passes on retry, so auto-resolve” is a trap, not a policy. A test that passes on retry is exactly what a race condition looks like — and exactly what a real, timing-dependent regression looks like too.

How do you wire it into CI without it becoming a gatekeeper?

Make it write-only to humans and read-only to the build. The triage layer posts a brief; it never changes the pass/fail gate, never reruns a “flaky” test as resolved, never closes a failure. The deterministic suite stays the gate. The LLM just makes the gate’s output legible.

On a suite I helped instrument at an anonymized fintech (roughly 600 specs, 16 workers, a four-person QA team), this cut median time-to-first-real-failure from about 70 minutes of log-reading to about 12 minutes — not because the model was smart, but because it reordered the queue so humans opened the right trace first. Zero failures were auto-resolved. The model never once said “flaky.” That restraint is the feature.

The connection to green tests that hide real bugs is direct: a green check mark is only as trustworthy as the process that produced it. An LLM that ranks and explains failures makes your green more trustworthy. An LLM that decides which failures don’t count makes it worthless.

What I’d do differently

The first version I built returned a verdict field with flaky | real | infra. It was seductive — the dashboard looked decisive. Within two weeks it had confidently tagged a genuine null-check regression as “flaky (likely timing),” and only a skeptical engineer who opened the trace anyway caught it before release. I ripped the verdict field out and replaced it with hypothesis + needs_human. Triage quality went up, because removing the model’s authority forced it to show its evidence, and showing evidence is what humans actually needed.

If I were starting today: ship the read-only reporter first, route the brief to Slack, and let it run for a month with humans still doing the deciding. Measure whether your time-to-real-failure drops. Only then consider deterministic auto-actions — and even then, only on signals you already trust deterministically, like quarantining a test that’s failed-on-retry more than N times across M runs. Never on the model’s opinion.

Your one concrete next step

This week, add a custom Playwright reporter that does nothing but collect result.error and result.attachments in onTestEnd and print a ranked Markdown summary at onEnd. No LLM call yet — just prove you can extract clean triage payloads from your own suite. That artifact is the entire foundation; the model is a 30-line addition on top of it. Get the payload right and the rest is wiring.