Test Failure Triage Has One Question: App or Test?

The failure count is the least useful number in your nightly report. It tells you how loud the night was, not what broke. Test failure triage should put one question to every red test: did the app change, or did the test system? Until each failure has that answer, the count is noise, and a falling count can hide a rising number of real defects.

You can answer that question with two mechanical checks: rerun the test alone, then run it against the last build where it passed. Neither needs judgment, and together they classify most failures in under ten minutes each.

Why is “how many failed” the wrong test failure triage question?

Because a count can’t be acted on, and a classification can. “28 failed” doesn’t tell a developer to look at anything, and it doesn’t tell an SDET what to fix. “3 app, 25 test” sends three failures to developers today and puts 25 on the QA backlog.

I learned this at a Canadian lottery and gaming operator. We ran about 1,100 Playwright and API tests nightly across four SDETs, and every standup opened with a number: “28 red, down from 41.” The trend line looked healthy for a month. During that month a payout-rounding defect sat in the red pile for nine days. Its test had been failing the whole time, but it was one row among 28, and the pile was shrinking, so nobody asked what had broken.

The fix was a rule rather than a tool. No failure leaves standup without a verdict: app or test. Anything still unclassified at standup becomes the day’s first job, and the unclassified count replaced the failure count on the dashboard. It’s the same reason pass rate misleads leadership: it measures the suite’s behaviour, not the product’s.

Why only two buckets: app or test?

Two buckets force an owner. Every failure goes either to a developer, because the app’s behaviour changed, or to the test team, because something in the machinery that produces the signal changed. That machinery includes test code, test data, environments, and CI runners.

Most triage schemes I’ve inherited had six categories: app, test, environment, data, flaky, unknown. “Environment” and “flaky” were where failures went to die, because nobody owned them.

”But environment failures aren’t test bugs”

They aren’t bugs in your spec file, fair enough. They are still failures of the system QA exists to run. If staging’s payment stub was down at 2 AM, the app didn’t regress; your signal did. Somebody has to own a reliable signal, and if it isn’t the test team, it’s nobody. You can tag these “test / env” for reporting. The verdict that sets ownership is still test.

What about intended app changes?

If a developer renamed the button on purpose, the verdict is still app. The behaviour changed, so the person who changed it confirms it; then the SDET updates the test. Skip that five-minute conversation and QA quietly decides what correct behaviour is, which is how a self-healing locator hid a real bug for three weeks.

How do you tell an app bug from a test bug in under 10 minutes?

Change one variable at a time. First, remove the rest of the suite by running the test alone. Then remove the new app build by running the same test against the last build where it passed. Whatever variable makes the failure disappear owns it.

Step 1: Isolate the failure

Run the failing test by itself, with no retries, several times in a single worker:

# Same test, alone, no retries, five times in one worker
npx playwright test tests/checkout/promo.spec.ts \
  -g "applies 10% promo to cart total" \
  --retries=0 --repeat-each=5 --workers=1 \
  --trace=retain-on-failure

Each flag here is in the Playwright CLI reference. Here’s how to read the result:

• Fails 5 of 5: the failure is deterministic. Go to step 2.
• Passes 5 of 5, but failed in the full run: verdict test. Something in the suite caused it, such as shared state, ordering, a parallel worker reusing an account, or data left behind by an earlier test.
• Mixed: it’s intermittent. Take the same --repeat-each into step 2 and compare failure rates between the two builds.

Step 2: Run it against last-known-good

Last-known-good (LKG) is the most recent app build where this test passed. Keep that build deployed somewhere, then point the same test commit at it. In Playwright, that means making baseURL overridable:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  use: {
    baseURL: process.env.BASE_URL ?? 'https://qa.example.internal',
  },
});

# Same test commit, same data seed, previous app build
BASE_URL=https://lkg.qa.example.internal npx playwright test \
  tests/checkout/promo.spec.ts -g "applies 10% promo to cart total" \
  --retries=0 --repeat-each=5 --workers=1

Then read it like a truth table:

The first row is the one that pays off: same test, data, and runner, and only the build differs. Developers don’t argue with that for long.

What does it take to keep a last-known-good build around?

A deployable artifact per green build and somewhere to run it. Teams with ephemeral Docker environments get this almost free: tag the image that last passed and start it next to the current build. At the gaming operator, we kept one “LKG” environment and redeployed it with whichever build passed the full nightly. That was one spare node and a script of about 40 lines.

If you can’t, use the inverse check: run the test commit from the last green run against the current app. If the old tests pass, your test change caused the failure. It’s weaker, since environment drift doesn’t show up in git, but it beats guessing.

How do you make classification the report instead of the count?

Generate a triage sheet automatically and leave the verdict column blank. This custom Playwright reporter writes one row per failed or flaky test:

import type { Reporter, TestCase } from '@playwright/test/reporter';
import { writeFileSync } from 'node:fs';

class TriageSheetReporter implements Reporter {
  private tests = new Set<TestCase>();

  onTestEnd(test: TestCase) { this.tests.add(test); }

  onEnd() {
    // outcome() is final only after all retries, so read it here
    const rows = [...this.tests]
      .filter((t) => t.outcome() === 'unexpected' || t.outcome() === 'flaky')
      .map((t) => {
        const err = t.results.at(-1)?.error?.message?.split('\n')[0] ?? '';
        return `| ${t.titlePath().filter(Boolean).join(' › ')} | ${t.outcome()} | ${err.replace(/\|/g, '\\|')} | ? |`;
      });
    const header = '| Test | Outcome | First error line | App or test? |\n|---|---|---|---|';
    writeFileSync('triage.md', [header, ...rows].join('\n'));
  }
}
export default TriageSheetReporter;

Register it with reporter: [['list'], ['./triage-sheet-reporter.ts']]. Flaky tests get a row too, because they still need a verdict. The ? column is the whole point: the only number you report is how many question marks are left. If you use an LLM to cluster CI failures, feed it this sheet, but have the two checks fill in the verdict.

After six weeks of this, roughly seven in ten failures were on the test side, and the app side had produced 23 real defects. Before, those defects would have sat in a pile we judged by its size. Triage dropped from the first 90 minutes of two people’s mornings to about 25 minutes for one person.

Where does this protocol break down?

It struggles when both sides changed at once, for example a release that ships an app change and a schema migration together, where the LKG build can’t run against the new data. In that case, rule out the test side first with the inverse check, and treat whatever survives as app until a developer shows otherwise. A false “app” verdict costs a developer ten minutes. A false “test” verdict ships a defect.

That asymmetry is my answer to “developers will get annoyed.” Some will, for about a week. After that, every failure that reaches them arrives with two runs differing in one variable, which beats most bug reports they’ve ever received.

Your next step

Tomorrow morning, pick the oldest red test in your nightly run. Run it alone with --retries=0 --repeat-each=5 --workers=1, then against the last build where it passed, and write down the verdict. If you can’t find or deploy that build, you’ve found the first thing to fix in your triage process.

FAQ