Don't Let AI Pick Which Playwright Tests to Skip

Your PR changes one CSS class and CI runs all 400 Playwright specs anyway. So someone wires up --only-changed, or worse, pastes the diff into an LLM and asks “which tests should I run?” Both feel smart. Both will eventually skip the exact spec that breaks — quietly, on the change that ships to prod.

I’ve spent a decade watching enterprise suites grow past the point where running everything on every push is defensible, and watching teams reach for test selection to fix it. The selection itself isn’t the mistake. The mistake is letting a heuristic — or a language model — narrow the test set with no one checking what it dropped. AI absolutely belongs in test impact analysis. Just not in the seat everyone’s putting it in.

Why test selection is suddenly worth the risk

Running the full suite used to be the safe default because it was cheap. It isn’t anymore. The average 50-person engineering team now spends roughly $3,000–$8,000 a month just on CI/CD compute, and that’s before you count the salaries burning while people wait.

So the pressure to run fewer tests is real and it’s financial. The question is never whether to select — it’s whether your selection method fails safe (runs a test it didn’t need to) or fails silent (skips a test it did). Get that backwards and you’ve traded compute dollars for shipped regressions, which is a terrible trade for any team that’s been burned once.

Why does Playwright’s --only-changed miss the test that breaks?

Because it selects on the static import graph, and a lot of what couples a test to your app never shows up in that graph. --only-changed (shipped in Playwright 1.46, still the built-in option through 1.61) runs the spec files you touched plus any spec that imports a changed file. That catches direct import edges and nothing else.

The fixture-injection edge the import graph can’t see

Here’s where it under-selects in a real suite:

// You edit this fixture to change how SSO sessions are minted.
export const test = base.extend<{ authedPage: Page }>({
  authedPage: async ({ page }, use) => {
    await loginViaSSO(page);
    await use(page);
    // Specs don't `import` this by name — they get it via the
    // test runner's fixture injection. The import graph can't see that edge.
  },
});

A spec that uses authedPage may never statically import the fixture file — Playwright injects it. Change the fixture and the dependency-graph heuristic can wave the SSO logout spec right through. The Playwright team is explicit about this: the docs call --only-changed a heuristic that “might miss tests” and tell you to run the full suite afterward anyway. Data files loaded by path, page objects resolved by string, config-driven test data — same blind spot.

What predictive test selection actually buys you

A correlation model fills the gap the import graph can’t. Instead of asking “what does this file import,” it asks “historically, when files like these changed, which tests actually failed?” That’s the Meta/Facebook Predictive Test Selection approach (arXiv 1810.05286), and the numbers are the reason it’s worth copying.

The model is a plain gradient-boosted decision tree trained on historical test outcomes — file paths changed, test, pass/fail — and, critically, it accounts for flakiness so a flaky red doesn’t poison the training signal. You don’t need Meta’s scale to start. A lightweight version is a table: for every CI run, log the changed paths and the specs that failed. After a few hundred runs you can rank specs by failure correlation to a given directory. That ranking is a far better selector than imports because it learns the runtime couplings the graph never sees — exactly the kind of nondeterministic failure I broke down in letting AI triage Playwright failures.

Where AI actually belongs: as the auditor, not the selector

Here’s my opinion, and it’s the whole point of this post: an LLM should never be the thing that narrows your test set. It should be the thing that catches a narrow set that’s wrong.

Why an LLM selector fails silent

Ask a model “which tests should I run for this diff?” and it will confidently return a list — including hallucinated relevance and, worse, confident omissions you can’t see. That’s under-selection with a friendly voice. But flip the job around and the model becomes genuinely useful: give it the diff, the already-selected set, and the dependency graph, and ask it to find what the selection missed.

The widen-only audit prompt

You are auditing a CI test-selection decision. Inputs:
- The git diff for this PR
- The list of specs the selector chose to run
- The repo's fixture + page-object map

Flag any changed behavior whose covering spec is NOT in the selected list.
Output only gaps. Widening the net is safe; do not propose removing tests.

The asymmetry is the safety mechanism. The model can only ever add tests back, never remove them. Its worst failure is running a spec you didn’t need — fail-safe, not fail-silent. When it spots that you edited the auth fixture but the SSO logout spec isn’t scheduled, it puts that spec back and tells you why. That’s AI doing what it’s good at — reading intent across messy context — without handing it a decision where a miss ships a bug. It’s the same line I draw with self-healing locators: assist the human, don’t replace the safety check.

The layered TIA pipeline I’d run in CI

Don’t pick one method. Stack them so each layer’s failure mode is covered by the next.

The backstop is non-negotiable. Selection is an optimization layered on top of a full run, never a replacement for it. Skip Layer 4 and the first time your model under-selects on a Friday deploy, you find out in production.

Those numbers track with what teams report from ordinary parallelization too — a 35-minute suite cut to ~10 by running wide. The difference here is you’re cutting what runs, not just spreading it across more workers, so the savings compound with the parallel work I covered in running parallel tests without refactoring and the broader make-your-suite-faster playbook.

How do you keep AI from hiding regressions in test selection?

Measure the miss rate before you trust any layer to skip a test. For your first month, run both the selected set and the full suite on every PR, and log every case where the full suite caught a failure the selection dropped. That miss rate is the only number that tells you whether your selection fails safe.

If you can’t measure the miss rate, you’re not doing test selection — you’re gambling with your release on the line, and calling it optimization.

The one thing to do this week

Add --only-changed=origin/main to your PR pipeline behind a full nightly run — but ship it in shadow mode first: keep running the full suite on PRs, and log every failure the selection would have skipped. After two weeks you’ll have a real miss rate, and you’ll know whether to trust selection at PR time or whether your suite’s coupling is too tangled for it yet. That measurement is worth more than any model.

Then, and only then, add the LLM as a widen-only auditor. Let it argue for more tests, never fewer. That’s the version of “AI in test selection” that makes your CI faster without making your releases a coin flip. For the rest of where I think AI does and doesn’t belong in a test suite, the AI testing pillar collects it.