Your Parallel AI Tests Fail on 429s, Not the Model

Your AI-feature specs pass every time on your laptop and fail a third of the time in CI, and you’ve spent two days blaming the model. Stop. Open one of the failing traces and read the status code: it’s a 429, not a bad answer. Your Playwright suite isn’t flaky — it’s eight worker processes firing live LLM calls into a shared tokens-per-minute quota at the same instant, and the provider is throttling you.

This is the failure mode nobody warns you about when you bolt an AI feature onto an existing regression suite. You already parallelize aggressively — that’s the whole point of Playwright workers — and every optimization you made to run the UI suite fast now works against you the moment those tests hit a rate-limited API. The stakes aren’t just red builds. Blind retries on a 429 quietly multiply your token bill, deepen the throttle with a retry storm, and — worst of all — can mask a real model regression behind an infrastructure error. Around 92% of Fortune 500 companies now run generative AI in production workflows, which means this is landing in your suite whether you planned for it or not.

Why does your AI suite 429 only in CI, not locally?

Because concurrency is the variable that changes between the two environments, and rate limits are a concurrency problem. On your laptop you run one spec, one call at a time. In CI, Playwright defaults to half the machine’s logical CPU cores — on a 16-vCPU runner that’s eight workers, each an independent process with its own browser and its own outbound API calls. Eight tests that each looked fine in isolation now hit the provider simultaneously.

The limit that actually bites is tokens per minute (TPM), not requests per minute. A single agentic test can burn thousands of tokens per call, so you blow the token ceiling long before you hit the request ceiling. OpenAI meters RPM and TPM separately, and Anthropic splits it further into input and output tokens per minute (ITPM/OTPM). One nuance worth knowing: for most Claude models, only uncached input tokens count toward ITPM — so prompt caching in your fixtures buys back real headroom.

Why doesn’t a p-limit semaphore fix it?

Because Playwright workers are separate OS processes, and an in-process semaphore only limits the process it lives in. This is the single most common wrong turn I see senior engineers take: they reach for p-limit or a shared async queue, cap it at four concurrent calls, and are baffled when the 429s continue.

import pLimit from 'p-limit';

// WRONG across a Playwright run. Each worker is its own OS process,
// so this module is re-imported per process and the counter resets.
// With 8 workers you get 8 × 4 = 32 concurrent calls to the provider,
// not the 4 you think you configured.
export const limit = pLimit(4);

The Playwright parallelism model is explicit that each worker has “its own memory” — which means any coordination primitive that lives in memory (a semaphore, a counter, a token bucket) is invisible to the other seven workers. Whatever caps your concurrency has to live outside the worker processes. Get this wrong and you’ll tune the semaphore for a week while the real limit sits at workers × yourLimit.

Why is retrying a 429 different from retrying model flake?

Because a 429 is a legitimate infrastructure signal you’re allowed to retry — but only if you honor the provider’s backoff, and only if you’re not laundering a quality regression in the process. This is the opposite of model flake, where blind retries hide the drop you most need to see. With a 429, the retry is fine; the fixed-interval retry is the problem.

Honor the provider’s retry-after, with jitter

Both providers hand you the exact wait time. OpenAI returns a retry-after-ms header on a 429; Anthropic returns a retry-after in seconds aligned to the specific limit you tripped. Read it instead of guessing, and add jitter — otherwise all eight workers wake up at the same millisecond and re-collide, the classic thundering herd that AWS documents as the reason plain exponential backoff isn’t enough.

export async function callModel(url: string, req: RequestInit, attempt = 0): Promise<Response> {
  const res = await fetch(url, req);
  if (res.status !== 429) return res;
  if (attempt >= 5) throw new Error('Rate limited after 5 backoffs — cut concurrency');

  // Trust the provider's number; fall back to exponential only if absent.
  const retryMs = Number(res.headers.get('retry-after-ms'))
    || Number(res.headers.get('retry-after')) * 1000
    || 2 ** attempt * 1000;
  const jitter = Math.random() * 250; // break the herd — workers must not re-collide
  await new Promise((r) => setTimeout(r, retryMs + jitter));
  return callModel(url, req, attempt + 1);
}

How do you actually cap concurrency across workers?

You move the limit to something all the workers share — because, as above, nothing in worker memory will do it. There are two approaches that work in practice, and they’re not mutually exclusive.

Split by project, throttle by invocation

The detail that trips people up: workers is a global Playwright option — you can’t set it per-project in the config. So you split by project for matching, and you throttle by invocation:

export default defineConfig({
  fullyParallel: true,
  projects: [
    { name: 'ui',      testMatch: /.*\.spec\.ts/ },              // deterministic, fast
    { name: 'ai-live', testMatch: /.*\.ai\.spec\.ts/, retries: 0 }, // every call is metered
  ],
});

# UI suite runs wide — no external quota to respect.
npx playwright test --project=ui --workers=50%

# Live AI lane runs narrow — cap processes UNDER your TPM headroom.
npx playwright test --project=ai-live --workers=2

This is the same instinct behind running parallel suites without a refactor tax and treating shared state as the enemy of parallel isolation — except here the shared resource isn’t a test user, it’s your provider quota. And the live lane belongs on a schedule, not the merge path: keep mocked model responses on every PR so the fast suite stays free and deterministic, and run the metered lane nightly where a pass@k reliability budget already wants a controlled sample anyway.

What I’d do differently

At a large payments company, we shipped an AI “explain this declined transaction” feature and wired the E2E checks into the existing suite the way we always had — full fullyParallel, twelve workers on the CI runner. The AI specs flaked on roughly 30% of runs. We spent the better part of a sprint instrumenting the model calls, tightening prompts, and arguing about temperature before someone finally grepped the traces and found every failure was a 429. Twelve workers were hitting a shared TPM ceiling, and our retries: 2 was re-spending tokens on each collision — we were paying more and getting less signal.

The fix took an afternoon once we understood it: a dedicated ai-live project pinned to two workers, a client that honored retry-after-ms with jitter, and the whole lane moved to a nightly graded job. Flake went to near-zero and the monthly token spend on CI dropped noticeably, because we stopped retrying our way through the rate limiter. What I’d do differently is obvious in hindsight — read the status code before you blame the model. An infra error dressed up as flakiness will eat a week if you let it.

The one thing to do today

Open your CI artifacts, find the last three “flaky” AI-test failures, and check the HTTP status on the model call. If any of them are 429, you don’t have a model problem — you have a concurrency problem, and no amount of prompt tuning will fix it. Split your live-model specs into their own Playwright project, run that project with --workers=2, and put retry-after handling in the API client instead of retries in the test. Then watch the flake rate — the same thing that looked like a green suite hiding real bugs usually turns out to be your own workers standing in line.

Want more field notes like this — the failure modes that only show up at enterprise scale? Grab the weekly QA automation newsletter using the signup just below. One battle-tested breakdown a week, no fluff, from 10+ years in the trenches.