Shuffle AI-Generated Playwright Tests on 4 Workers, Not 1

I had Claude Code write three Playwright suites for the same small app, then ran the AI-generated Playwright tests with Playwright 1.64’s --shuffle on a single worker: 50 shuffled runs, zero failures. With the same seeds and four workers, 21 different tests broke.

So shuffle with real parallelism, then use isolated retries to confirm interference and test locks to contain it. In my demo, single-worker --shuffle only tested what all three agents had already handled: cleaning up after themselves in sequence. What two of them got wrong was concurrency, and that only shows up when tests share the server with other tests running at the same time.

Why do AI-generated Playwright tests hide shared-state bugs?

An agent checks its suite against the config in front of it. If the config says workers: 1, “the suite passes” only means “the suite passes when nothing else runs at the same time”. The agent has no reason to test anything else, and it may not know the newer Playwright flags exist.

Both effects showed up in my demo. One suite’s agent ended its run with this, quoted from its own summary:

It doesn’t prove the tests work in any order, since Playwright can’t shuffle tests natively and I didn’t run them shuffled.

That was out of date. --shuffle shipped in Playwright 1.64.0 on October 7, 2026. The PR that added it (#42877) lists issue #9297, “Run tests in a random order”, opened in 2021, as fixed. A second agent needed tests to take turns on the global settings. It wrote its own cross-worker lock: a temp directory created with mkdirSync, a polling loop and a 15-second stale-lock timeout. Playwright 1.63 already had that built in as one option on the test.

This is the gap to close before you freeze the generated specs into your regression suite. Freezing makes the suite deterministic, but it also locks in any coupling the agent left behind.

The demo: three AI-generated suites, one shared server

The fixture app is a 125-line Node task tracker. All its state lives in memory: a task list, an open-task counter, and global settings (display name and “show completed tasks”). There is no reset endpoint, which is common in shared QA environments. Every suite got the same prompt: cover listing, adding, editing, completing, deleting, the counter and settings, with about 20 tests, and iterate until npx playwright test passes.

All three passed in default order on the first try. All code and logs are from Playwright 1.65.0 on a single Mac.

That’s one generation per suite, so read the results as what these three suites did, not as a rate for AI-generated tests in general. A rerun of the same prompt could produce a suite with different coupling, or none.

What does Playwright’s --shuffle actually shuffle?

It shuffles test files. It shuffles individual tests only when they run in parallel mode, either through fullyParallel or with --fully-parallel on the command line. Serial suites stay together and in order. The seed is printed at the start of the run, and you pass it back to reproduce the order (CLI docs).

# random seed, printed in the header
npx playwright test --shuffle
# Running 25 tests using 8 workers, shuffle seed 7

# replay an order, and shuffle inside files too
npx playwright test --shuffle 7 --fully-parallel --workers=4

There is no shuffle option in playwright.config.ts. It is CLI-only, so it lives in your CI job rather than in the config. The flag also takes an optional value, and that causes a real trap:

Running 24 tests using 1 worker, shuffle seed tests/settings.spec.ts

Playwright read the file path as the seed and dropped the filter, so it ran all 24 tests instead of 5. It was reported as issue #43156 and closed without a fix, so it is still the 1.65 behavior. Put file filters before --shuffle, or always pass an explicit seed.

Why did shuffling AI-generated Playwright tests on one worker find nothing?

Because these three agents had already handled order dependence. Every test created tasks with unique titles, never touched the three seed tasks, and restored settings when it finished. That made the suites safe to run in any sequence. It did not make them safe to run alongside other tests, because a test that is running can’t clean up state that another worker is changing.

I ran each suite under 10 fixed seeds per configuration:

Suite A had been written for parallel runs and stayed green at every worker count. Most of suite B’s failures were delta assertions on the global counter: read the counter, act, then expect before + 1. Another worker added a task in between, and the count was off by one. The rest were settings collisions, such as one test’s afterEach resetting the display name while another test asserted its new one. Suite C’s beforeEach deleted every task it didn’t recognize, including tasks other workers had created a moment earlier.

The coupling in these suites was not “test X needs test Y to run first”. It was “test X needs nobody else touching the counter”. Single-worker shuffle doesn’t test that, however many seeds you run. In a shared QA environment it is worse than my demo: other teams’ pipelines and manual testers act as extra workers you don’t control, so four local workers understate the contention.

Shuffled versus default order at four workers

Shuffling still helps once the workers are there. In ten unshuffled runs at four workers, suite C surfaced 9 distinct failing tests. Ten shuffled runs surfaced 14. In the unshuffled runs, the same two tests failed every time and the rest rarely showed up. Shuffling changes which tests run at the same time, so more of the bad pairings get a chance to happen. For suite B, both modes found 7 distinct tests, but not the same 7. Each mode caught one that the other missed.

A seed replays the order, not the timing

I ran suite B with --shuffle 1 at four workers five times. The runs had 3, 4, 4, 4 and 4 failures, and the set of failing tests changed between runs. The seed fixes the order in which tests are handed out. It does not fix when each worker picks up its next test. Use the seed to reproduce a failure, and expect to rerun it a few times. Don’t treat a passing replay as proof that the bug is gone.

How do isolated retries tell interference from a broken test?

retryStrategy: 'isolated' (Playwright 1.62) holds every retry until the rest of the run has finished, then runs the retries one by one in a single worker (testConfig docs). A test that fails in the crowd and passes alone is failing because of interference. A test that also fails alone is broken.

import { defineConfig } from '@playwright/test';
import base from './playwright.config';

export default defineConfig(base, {
  retries: 1,
  retryStrategy: 'isolated', // retries run last, one by one, in a single worker
  failOnFlakyTests: true, // "passed on retry" still fails the job
});

I ran suite B across the same 10 seeds at four workers:

• With the default 'immediate' retries, the 10 runs still ended with 17 hard failures and 18 flaky results. A retry that runs while three other workers are busy can hit the same collision again.
• With 'isolated', there were 0 hard failures and 37 flaky results. Every failure passed once it ran alone.

Suite C came out the same way: 18 hard failures with immediate retries, 0 with isolated. That points at shared state. It also explains why the healer agent tends to hide race conditions: the evidence goes green as soon as the test runs alone.

How do you fix shared-state tests with Playwright test locks?

Give every test that reads or writes the same global state the same named lock, and Playwright won’t run those tests at the same time (docs). If a test asserts on something every other test changes, like a global counter, use the built-in -exclusive- lock from 1.65. It runs the test only when nothing else is running.

For suite B, the fix took eight annotations: seven -exclusive- tests and one settings lock on the settings describe block.

test.describe('Open-task counter', () => {
  test('increments when a task is added', async ({ page }) => {
  test('increments when a task is added', { lock: '-exclusive-' }, async ({ page }) => {
    await page.goto('/');
    const before = await openCount(page);
    await addTask(page, uniqueTitle('Counter add'));
    expect(await openCount(page)).toBe(before + 1);
  });

test.describe('Settings page', () => {
test.describe('Settings page', { lock: 'settings' }, () => {

The locked suite then passed 30 shuffled runs in a row, 10 at four workers and 20 at eight. It finished in about 2.8 seconds, against 4.4 seconds for the original single-worker run. That is a smaller speedup than four workers suggests, because seven of the 24 tests now run alone.

My first pass had seven annotations, and that turned up a test I would have missed. Once the locks changed the scheduling, “new tasks are appended to the end of the list” began failing in 3 of the next 20 runs. It checks that the last row on the page is the task it just created, and another worker sometimes added a task first. That race had never shown up in the 20 unlocked four-worker runs before it, which is the usual shape of a flaky test that is really a race condition. So after you add locks, run the full seed matrix again rather than one confirmation run.

Suite A’s fix went the other way. I deleted the agent’s mkdir lock, with its polling, stale-lock cleanup and try/finally, and declared the lock on each test instead:

export const test = base.extend<{ settingsLock: void }>({
  // Tests that use this fixture must also declare { lock: 'settings' }.
  settingsLock: async ({ request }, use) => {
    await saveSettings(request, DEFAULT_SETTINGS);
    await use();
    await saveSettings(request, DEFAULT_SETTINGS);
  },
});

That change removed 49 lines and added 12, and suite A passed 20 more shuffled runs. Unlike the temp directory, a Playwright lock can’t be left behind by a crashed run. Waiting for a lock doesn’t count against the test timeout (PR #41862).

When a test lock can’t save the suite

Suite C’s beforeEach deletes every task that isn’t a seed task. That affects every other test in the run, so every test would need -exclusive-, which is the same as workers: 1 with extra steps. Locks only help when few tests touch the shared state. When the agent’s isolation strategy is “reset the world”, the fix is to scope the data to each test. That is the same move as giving up shared test users racing each other, applied to the backend instead of the account.

The cheapest fix came before generation. I gave Haiku the same prompt with one extra line: the suite must also pass with npx playwright test --fully-parallel --workers=4 --shuffle. The new 22-test suite started its own copy of the app on a free port for each test, and passed all 10 shuffled seeds at four workers. Starting a server per test only works for a toy app. The useful part is that one sentence in the prompt changed how the agent approached isolation. That’s a single generation, so it’s a cheap experiment worth repeating on your own suite, not a guarantee.

How do you add a shuffle probe to CI?

Run a probe like this nightly on any suite that an agent wrote or changed. It runs random seeds at real parallelism and prints each seed so a failure can be replayed:

#!/usr/bin/env bash
# Run the suite under several random shuffle seeds with real parallelism.
# Exit non-zero if any seed fails, and print the seeds so failures can be replayed.
WORKERS=${WORKERS:-4}
RUNS=${RUNS:-10}
status=0
for i in $(seq 1 "$RUNS"); do
  seed=$RANDOM
  log="shuffle-$seed.log"
  if npx playwright test --fully-parallel --workers="$WORKERS" --shuffle "$seed" --reporter=dot > "$log" 2>&1; then
    echo "seed $seed: passed"
  else
    echo "seed $seed: $(grep -Eo '[0-9]+ failed' "$log" | head -1)"
    status=1
  fi
done
exit $status

On suite B before the locks, four seeds gave 5, 3, 5 and 5 failures, and the script exited 1. After the locks, all four seeds passed and it exited 0. Set WORKERS higher than your CI’s normal worker count. You want to catch the coupling before someone raises the worker count to speed up the pipeline. Keep the probe out of the PR gate: ten runs of a large suite cost too much there, and the seeds give you everything you need to reproduce a failure the next morning.

Surviving the probe is evidence of independence, not proof: the append-order race above hid through 20 unlocked runs. Whether the tests assert anything is a separate check, and mutation testing tells you whether they do.

Do this before you trust the next generated suite

Add one line to your generation prompt or AGENTS.md: “The suite must pass with npx playwright test --fully-parallel --workers=4 --shuffle.” Then run shuffle-probe.sh against whatever comes back. If anything fails, rerun it with an isolated-retry config to confirm interference, then add locks only where the shared state really can’t be scoped to a single test.