Your Playwright Tests Race the AI Token Stream

Your AI chat feature streams its answer token by token, and your Playwright test asserts on the text the instant it appears. That assertion passes — against a half-rendered answer — and the build goes green. Then a downstream check fails one run in five because it fired while the model was still typing. The bug isn’t your assertion. It’s that Playwright’s auto-wait considers a mid-stream DOM “stable” and hands you a partial answer with a straight face.

This is a different problem from asserting the contract instead of the prose. That post is about what to assert on a non-deterministic answer. This one is about when — the synchronization race between Playwright’s actionability model and a response that arrives in fragments over a few seconds. You can have a perfect semantic assertion and still flake, because you ran it against the first 40 tokens of a 400-token reply.

Why does Playwright’s auto-wait fail on streaming responses?

Because auto-wait checks for DOM stability between animation frames, and a token stream is stable between most frames. Tokens arrive in bursts every 50–150ms; in the gaps, the element isn’t changing, so Playwright’s actionability checks consider it settled and your assertion resolves — on whatever text happened to be there in that gap.

Server-Sent Events are the transport underneath nearly all of this. SSE is the de facto standard for LLM streaming, used natively by OpenAI and Anthropic, precisely because it pushes incremental chunks over plain HTTP. Each chunk appends a few tokens to the DOM node. Playwright’s toContainText with a substring match is the trap: it retries until the substring appears, which can be true at token 30 even though the meaningful content lands at token 380.

I’ve watched this exact failure ship at two orgs in the last year. A product team adds a streaming “explain this” panel, QA writes the test the way they always have, and within a sprint the suite has a flake everyone learns to re-run past. That’s the real cost — not the red build, but the moment your team stops trusting the suite and treats a green check mark as noise that hides real bugs.

What does a flaky streaming test actually look like?

Here’s the naive version. It looks correct, passes locally every time you run it by hand, and flakes in CI under load when the stream is slower.

test('chat answers a refund question', async ({ page }) => {
  await page.goto('/support');
  await page.getByRole('textbox', { name: 'Ask a question' }).fill('How do I get a refund?');
  await page.getByRole('button', { name: 'Send' }).click();

  // Resolves on the FIRST frame containing "refund" — mid-stream.
  await expect(page.getByTestId('assistant-message')).toContainText('refund');

  // Flakes: the Copy button only renders after the stream closes.
  await page.getByRole('button', { name: 'Copy answer' }).click();
});

The toContainText line is the silent failure. The Copy answer click is the loud one — it throws intermittently because that button is rendered only when the response completes, and the test got there first. Most teams “fix” this by adding { retries: 2 } and never look again. Retries don’t fix a race; they just roll the dice until it lands your way, while the underlying mid-stream assertions keep passing on partial content.

How do you wait for an AI stream to actually finish?

Wait for an explicit completion signal, not for the content. The single most reliable fix is to make the frontend expose when streaming is done and assert on that state transition before you touch the answer text.

<div
  data-testid="assistant-message"
  data-streaming={isStreaming ? 'true' : 'false'}
>
  {text}
</div>

await page.getByRole('button', { name: 'Send' }).click();

const message = page.getByTestId('assistant-message');

// Deterministic: this only passes once the SSE 'done' event fired.
await expect(message).toHaveAttribute('data-streaming', 'false');

// Safe to assert the contract now — the answer is whole.
await expect(message).toContainText('refund');
await page.getByRole('button', { name: 'Copy answer' }).click();

What if you don’t control the frontend markup?

Then wait on the network boundary, or poll for text stability. When the app is owned by another team and you can’t get a data-streaming attribute this quarter, you have two fallbacks that don’t depend on the DOM telling you the truth.

Fallback 1: Wait for the SSE response to close

The cleaner fallback is to wait for the SSE response to close. The stream is one long-lived HTTP response; when it ends, the answer is complete by definition.

// Start waiting BEFORE the click so you don't miss the response.
const streamDone = page.waitForResponse(
  (res) => res.url().includes('/api/chat') && res.status() === 200,
  { timeout: 30_000 },
);
await page.getByRole('button', { name: 'Send' }).click();
await streamDone; // resolves when the SSE connection closes

await expect(page.getByTestId('assistant-message')).toContainText('refund');

Fallback 2: Poll until the text stops changing

The scrappier fallback, when even the network is opaque, is a stability poll: wait until the answer’s text length stops growing across a quiet window. It’s uglier, but it encodes the real definition of “done” — the content stopped changing.

export async function waitForStableText(locator, quietMs = 800) {
  let previous = '';
  await expect
    .poll(async () => {
      const current = await locator.innerText();
      const stable = current === previous && current.length > 0;
      previous = current;
      return stable;
    }, { timeout: 30_000, intervals: [quietMs] })
    .toBe(true);
}

I reach for the network-boundary approach by default and keep the stability poll as the last resort. The poll has a real failure mode: a model that pauses mid-stream for longer than quietMs will trip it early. I’ve been burned by that on a reasoning model that streamed its answer, went quiet for a full second while it “thought,” then resumed. Tune quietMs above your worst observed inter-chunk gap, and never below 500ms.

Why not just mock the stream and skip the race entirely?

Because you should — for the PR gate — but the race still exists in the layer you can’t mock away. The right architecture is two lanes: mock the LLM in CI, eval live nightly. When you mock the SSE endpoint, you control the chunk cadence, so you can make the stream deterministic and fast.

But here’s the subtle part enterprise teams miss: a mocked stream that fires all its chunks in one synchronous burst hides the exact bug you’re trying to catch. If your mock doesn’t preserve realistic inter-chunk delays, your “Copy answer” button race never reproduces in CI — it only shows up in production, where the network is slow. Mock the content deterministically, but keep the timing realistic.

A minimal route mock that preserves cadence keeps the synchronization logic honest:

await page.route('**/api/chat', async (route) => {
  const chunks = ['Yes, ', 'you can ', 'request ', 'a refund ', 'within 30 days.'];
  const body = chunks.map((c) => `data: ${JSON.stringify({ delta: c })}\n\n`).join('');
  // The blank line + small delay matters: it forces the same
  // append-per-chunk DOM churn the streaming gate has to survive.
  await new Promise((r) => setTimeout(r, 60 * chunks.length));
  await route.fulfill({
    status: 200,
    headers: { 'content-type': 'text/event-stream' },
    body: body + 'data: [DONE]\n\n',
  });
});

What’s the rule that makes streaming AI tests boring?

Separate the completion gate from the content assertion, always, and never let a substring match double as your “is it done” signal. Once those are two distinct steps — wait for done, then assert the contract — streaming AI tests stop being special. They’re just async UI tests with an explicit ready state, the same pattern you’d use for a report that renders after a spinner disappears but is only half-painted.

Your one concrete next step today: grep your AI-feature specs for toContainText and toHaveText calls that run before any explicit wait on stream completion. Every one of those is a latent mid-stream race. Add a data-streaming flag (or an aria-busy toggle) to the component, gate on it, and move the content assertion after the gate. That single refactor will retire more “intermittent” AI test failures than any number of retries.