Your Parallel AI Tests Are Racing the Prompt Cache

Your AI feature’s Playwright suite passes on your laptop and flakes in CI, and every failure is a timeout waiting for the first token to render. You’ve already blamed the model, bumped the timeout, added a retry. None of it is the bug. The bug is that you turned on prompt caching to cut your inference bill, and prompt caching is a shared, time-decaying, concurrency-hostile piece of state that your test suite now depends on and does not control.

Here’s the whole thing in one line: a cache entry does not exist until the first response begins, so when eight Playwright workers fire the same prompt prefix at the same instant, all eight miss. The first request pays the cold-path latency. So do the other seven. Then, on your laptop, running one test at a time, every request after the first one hits a warm cache and returns in a fraction of the time — which is exactly why you tuned your timeouts against a reality CI will never reproduce.

Why do parallel AI tests miss the prompt cache?

Because cache writes are not atomic with request dispatch. Anthropic’s documentation says it plainly: “a cache entry only becomes available after the first response begins. If you need cache hits for parallel requests, wait for the first response before sending subsequent requests.” That single sentence is a design constraint on your entire test architecture, and almost nobody reads it as one.

Play it out with a fullyParallel Playwright project and workers: 8. All eight workers boot, all eight navigate, all eight trigger a chat request carrying the same 6,000-token system prompt and tool definitions. Nothing is cached yet. Eight cold requests go out in the same 200ms window. Eight cache writes happen — at 1.25× the base input token price — and zero cache reads. Your suite just paid ten times what it should have (cache reads are billed at 10% of base input) and every one of those eight tests saw worst-case time-to-first-token.

A latency delta that large is not a rounding error you can absorb with a slightly bigger timeout. It’s the difference between a 400ms first token and a 2.5s first token on the same prompt with the same model. If your test asserts that a streaming skeleton appears within 1 second, that test is a coin flip whose bias is set by scheduling order — the definition of a flaky test. And flakiness is already the dominant tax on suites: the share of teams reporting flaky tests climbed from 10% in 2022 to 26% in 2025. Adding a shared cache to a parallel suite is volunteering to grow that number.

The routing trap nobody tells you about

It gets worse on OpenAI, and this is the detail that took me longest to find. OpenAI’s caching is automatic for prompts of 1,024 tokens or more, and requests are routed to a machine by a hash of the prompt’s initial prefix — the cache is per-machine, not global. You can influence that routing with prompt_cache_key. And there’s a throughput ceiling: roughly 15 requests per minute per prefix + prompt_cache_key combination before requests start spilling onto other machines, where the cache is cold.

Read that against a CI suite. Sixteen workers × several model calls per test is trivially north of 15 rpm on a single shared prefix. Your requests get scattered across inference machines, hit rate collapses to something between 0% and “depends on the day,” and your latency distribution grows a fat tail — the exact behavior that tail-optimized caching research exists to characterize. Meanwhile your dev loop, running one test at a time, sits comfortably under the ceiling and gives you a 100% hit rate and a beautiful, lying benchmark.

How do you make the prompt cache deterministic in CI?

You stop letting the cache be an emergent property of scheduling order and make it an explicit setup step. Three moves, in order.

Pre-warm once, then let the workers read

import Anthropic from '@anthropic-ai/sdk';
import { SYSTEM_PROMPT } from './fixtures/prompt';

/** Runs once, before any Playwright worker exists. */
export default async function globalSetup() {
  const client = new Anthropic();

  // max_tokens: 0 writes the cache entry and returns immediately —
  // no output tokens billed, no response to throw away.
  const warm = await client.messages.create({
    model: 'claude-sonnet-5',
    max_tokens: 0,
    system: [{ type: 'text', text: SYSTEM_PROMPT, cache_control: { type: 'ephemeral' } }],
    messages: [{ role: 'user', content: 'warmup' }],
  });

  if (warm.usage.cache_creation_input_tokens === 0) {
    throw new Error('Cache write did not happen — prefix is under the model minimum.');
  }
}

That guard on the last line matters more than it looks. The minimum cacheable prompt is model-dependent — 1,024 tokens on Sonnet 5 and Opus 4.8, 4,096 on Haiku 4.5 — and if you’re under it, nothing is cached and no error is returned. Your suite runs cold and silent, and the only evidence is a bill. Fail loudly in setup instead.

Assert on the cached-token count

Now the actual test asserts a fact instead of a stopwatch:

import { test, expect } from '@playwright/test';

test('summarize hits the warm system-prompt cache', async ({ page }) => {
  const usage = page.waitForResponse((r) => r.url().includes('/api/summarize'));

  await page.goto('/documents/quarterly-report');
  await page.getByRole('button', { name: 'Summarize' }).click();

  const body = await (await usage).json();

  // Deterministic. Survives parallelism. Catches prompt refactors
  // that silently drop the hit rate to zero and 10x the bill.
  expect(body.usage.cache_read_input_tokens).toBeGreaterThan(1000);
  expect(body.usage.cache_creation_input_tokens).toBe(0);

  await expect(page.getByTestId('summary')).toBeVisible();
});

This is the assertion I’d add first if I could only add one. It’s a cost regression test, and it catches a whole class of bug that no functional test will ever see: someone appends a Generated at ${new Date()} line to the system prompt, or reorders the tool definitions, or flips tool_choice, and the prefix hash changes. Cache hit rate goes to zero. Every test still passes. Every response is still correct — Anthropic is explicit that caching has no effect on output token generation, the response is byte-for-byte what you’d get uncached. The only symptom is a 10× input bill that shows up three weeks later in the token budget you set for the feature.

Force the cold path instead of hoping for it

Separate the two behaviors into two projects. One warm, parallel, cheap. One cold, serial, deliberate.

export default defineConfig({
  globalSetup: './tests/global-setup.ts',
  projects: [
    {
      name: 'ai-warm',
      // Every worker reads the entry globalSetup wrote. Parallel is safe
      // because nothing here writes to the cache.
      fullyParallel: true,
      testDir: './tests/ai/warm',
    },
    {
      name: 'ai-cold',
      // A nonce in the prefix guarantees a miss. Serial, because a cold
      // stampede is exactly the thing we're testing one instance of.
      workers: 1,
      fullyParallel: false,
      testDir: './tests/ai/cold',
    },
  ],
});

The cold project is where your latency assertions live, and they’re honest there, because you know for certain the cache is cold. Assert your p95 cold TTFT budget in ai-cold and your cached-token counts in ai-warm. Neither test can lie to you about which world it’s in.

What this pattern doesn’t fix

Pre-warming makes the cache deterministic; it doesn’t make it free, and it doesn’t extend the TTL. The default 5-minute window is refreshed on every hit, so a busy suite keeps it alive on its own — but a slow lane, a queued runner, or a long beforeAll can easily leave a 5-minute gap, and then your “warm” project quietly runs cold. If your suite has gaps that long, pay for the 1-hour TTL for the CI lane or re-warm between projects.

And this is orthogonal to model behavior. A warm cache changes your latency and your bill, not your output — which means it cannot save you from a model that got worse. That’s a different gate entirely: see the real-model drift eval. Rate limits are also a separate failure mode with a separate fix, though they show up in the same 8-worker lane; I covered that in handling 429s in parallel AI tests.

Your one next step this week

Open the single AI test you trust least — the timeout-flaky one — and add one line: expect(body.usage.cache_read_input_tokens).toBeGreaterThan(0). Run it with your normal worker count. If it fails, you just proved your suite has been paying cold-path prices and cold-path latency on every run, and you now know exactly why that test flakes. Then add the globalSetup warm request and watch both the flake and the bill drop in the same commit.