Mock LLM Responses in CI, Eval Nightly: Two-Lane Tests

Your AI-feature end-to-end tests call a real model on every pull request. That’s why they’re slow, occasionally red for no reason, and quietly billing your team for the privilege. The fix isn’t a cleverer assertion — it’s a network boundary. You mock LLM responses in Playwright so the PR gate is deterministic and free, then run a separate live eval on a schedule to catch the drift the mocks can’t see.

I’ve shipped this two-lane setup at two enterprises now, and the reaction is always the same: people assume mocking the model means you stop testing the AI. It’s the opposite. Mocking is what lets you test everything around the model deterministically — the rendering, the error states, the retry logic, the empty-result UI — while moving the genuinely non-deterministic question (“is the answer any good?”) into a lane built to handle non-determinism.

Why shouldn’t CI call the real LLM on every PR?

Because a live model call taxes you three ways at once, and a PR gate is the worst place to pay all three. It adds cost (every run burns tokens), latency (model round-trips are seconds, not milliseconds), and non-determinism (the same prompt returns different text, so the same test passes and fails on identical code).

The cost is not theoretical. LLM API pricing in 2026 runs from roughly $0.10 per million input tokens for budget models up to about $30 per million for premium tiers, per CloudZero’s pricing survey. A chat feature with a 4K-token system prompt, called across a few dozen E2E specs, run on every push by every engineer, plus retries — that’s a line item your finance team will eventually ask about. I’ve watched a 30-spec AI suite quietly cost more in model tokens than the entire CI compute bill it ran on.

The non-determinism is the part that actually erodes the team. A test that calls a live model and asserts on its output will flake, and a flaky gate trains everyone to hit “re-run” on red. That’s how you end up with a green check mark that’s hiding a real regression — the build went green on the second roll of the dice, not because the code is correct.

The two-lane model: mock on the PR gate, eval nightly

The core move is to stop asking one test suite to do two incompatible jobs. Verifying your UI wiring wants determinism; verifying answer quality requires sampling a non-deterministic system. Split them into two lanes with different rules.

Why two lanes instead of one tuned suite?

Because no single configuration satisfies both jobs. Lane 1 is the gate that blocks merges, so it must never flake: every request to the model is intercepted and answered from a fixture. Lane 2 is where you let real models run, measure quality with thresholds instead of exact strings, and alert when scores slip — without ever standing between an engineer and their merge. This is the same instinct behind keeping your suite fast by pushing slow work off the critical path; the model call is just the slowest, priciest dependency you’ve got.

How do you mock an LLM response in Playwright?

You intercept the request the app makes to the model with page.route() and answer it from a fixture using route.fulfill(). Nothing about the model being an LLM changes the mechanics — it’s an HTTP call, and Playwright treats it like any other XHR or fetch.

Start with the skeleton. Match the endpoint your app actually hits (your own /api/chat proxy, or the provider URL if you call it directly from the browser):

import { test, expect } from '@playwright/test';

test('renders the summary returned by the model', async ({ page }) => {
  await page.route('**/api/chat', async (route) => {
    await route.fulfill({ path: 'tests/fixtures/summary.json' });
  });

  await page.goto('/documents/42');
  await page.getByRole('button', { name: 'Summarize' }).click();
});

Now make the assertion test the wiring, not the prose. The fixture is your contract: because you control the bytes the model “returns,” you can assert exact text, exact counts, exact IDs — the things that flake against a live model become rock-solid against a fixture.

await page.route('**/api/chat', async (route) => {
  await route.fulfill({
    contentType: 'application/json',
    // The fixture is the contract: a known answer with a known shape.
    json: {
      summary: 'Q3 revenue rose 12% on strong EMEA demand.',
      citations: [{ docId: 'fin-q3', page: 4 }],
      tokensUsed: 318,
    },
  });
});

await page.getByRole('button', { name: 'Summarize' }).click();
await expect(page.getByTestId('summary-text')).toHaveText(/rose 12%/);
await expect(page.getByTestId('citation')).toHaveCount(1);

The leverage here is that you can now test the cases a live model rarely produces on demand. Want to verify the empty-state UI? Fulfill with an empty summary. The error banner? Fulfill with status: 500. The token-budget warning? Set tokensUsed to something huge. These are the states that ship broken because nobody could reliably reproduce them — and with a mock they’re one fixture each.

How do you mock a streaming (SSE) LLM response?

For a streamed chat UI, fulfill the request with Content-Type: text/event-stream and a fixture containing the raw Server-Sent Events frames. The catch — and this is the part the docs gloss over — is that route.fulfill() delivers the whole body in one shot, so you can assert the final text but you are not actually testing the token-by-token streaming experience.

Here’s the static replay, which covers most needs. The fixture mirrors what your backend emits, including the data: prefixes and the [DONE] sentinel that OpenAI-style endpoints send (per OpenAI’s streaming format):

data: {"choices":[{"delta":{"content":"Q3 revenue "}}]}

data: {"choices":[{"delta":{"content":"rose 12%."}}]}

data: [DONE]

await page.route('**/api/chat', async (route) => {
  await route.fulfill({
    contentType: 'text/event-stream; charset=utf-8',
    path: 'tests/fixtures/chat-stream.txt',
  });
});

When the static replay isn’t enough

When you genuinely need to test progressive rendering — the cursor blink, the “stop generating” button mid-stream, partial-markdown handling — fulfill won’t get you there, because the browser receives everything at once. The durable approach is to stub the streaming primitive itself with context.addInitScript(): replace EventSource (or the fetch/ReadableStream reader your SDK uses) with a mock you drive token-by-token from the test. Microsoft’s maintainers point to exactly this pattern, and it’s worth the setup only for the handful of specs that assert on streaming behavior rather than streaming content.

// Injected via context.addInitScript — drives the stream from the test.
class MockEventSource {
  static instances: MockEventSource[] = [];
  listeners: Record<string, (e: { data: string }) => void> = {};
  constructor(public url: string) { MockEventSource.instances.push(this); }
  addEventListener(type: string, cb: (e: { data: string }) => void) {
    this.listeners[type] = cb;
  }
  emit(token: string) { this.listeners['message']?.({ data: token }); }
  close() {}
}
(window as unknown as { EventSource: unknown }).EventSource = MockEventSource;

That distinction — content versus behavior — is the senior call here. Ninety percent of your streaming specs only care about the final answer and belong on the cheap static-replay path. Reserve the init-script machinery for the few that test the stream as an interaction.

How do you keep fixtures from rotting?

Record them from a real response once, replay them forever, and re-record on demand. Hand-written fixtures drift away from what the model and your backend actually produce; recorded ones start accurate and stay honest because regenerating is a one-flag operation. Gate it behind an environment variable so the default path is always replay.

import { test } from '@playwright/test';
import fs from 'node:fs';

export async function routeLLM(page, urlGlob: string, fixture: string) {
  await page.route(urlGlob, async (route) => {
    if (process.env.RECORD) {
      const response = await route.fetch();          // hit the real model
      fs.writeFileSync(fixture, await response.text());
      return route.fulfill({ response });
    }
    await route.fulfill({ path: fixture });          // replay the recording
  });
}

This record/replay split is the same hygiene that makes storage-state contamination avoidable: pin the external state your tests depend on, and regenerate it through a controlled, reviewable step instead of letting it leak in live.

What does the nightly eval lane actually check?

It answers the one question mocks structurally cannot: is the model still producing good answers? Mocks freeze a known-good response in time, so they’ll happily stay green while the live model regresses after a provider update or a prompt tweak. The nightly lane hits the real API, samples it, and scores quality with thresholds instead of exact strings.

test('refund answers stay on-topic across runs', async ({ request }) => {
  let passes = 0;
  for (let i = 0; i < 5; i++) {                       // sample, don't single-shot
    const res = await request.post('/api/chat', {
      data: { prompt: 'Where is my refund?' },
    });
    const { summary } = await res.json();
    if (/refund|return|processed/i.test(summary)) passes++;
  }
  expect(passes).toBeGreaterThanOrEqual(4);            // 4-of-5, a measured bar
});

Run this on a schedule and before releases, never on the PR gate. Use an m-of-k pass rate so you’re measuring the distribution rather than gambling on a single roll, and when you graduate to scoring nuanced qualities like tone or groundedness, treat the model-judge as a noisy gauge with a threshold rather than an oracle. This lane is also where token cost is justified — the kind of spend that’s reckless on a PR gate but reasonable on a nightly job, which is exactly the cost-versus-context tradeoff behind picking an AI integration for 2026. The point of Lane 2 is to fail loudly and off the critical path when quality drifts, so no one is ever blocked by a model having an off day mid-merge.

What I’d do differently

The first AI feature I tested, I put everything in one lane: real model, every PR, exact-text assertions. It worked for about a week. Then the flakes started, the token bill showed up, and a “stable” feature spec turned into the one everybody skipped. I spent days adding retries, which only made it slower and hid the regressions I actually needed to see.

What I’d do now, from the first commit: draw the network boundary on day one. Mock the model for everything UI-facing, record fixtures from a real call so they start accurate, and stand up the nightly eval lane before the feature ships — not after it starts flaking. The teams that separate “does the wiring work” (deterministic, mocked, blocking) from “is the answer good” (sampled, live, non-blocking) end up with AI suites they trust. The teams that don’t end up with retries: 3 and a slow erosion of faith in every green build.

Your one concrete next step

This week, take your slowest AI-feature spec — the one that calls a real model — and wrap its model call in a single page.route() that fulfills from a fixture you record once with RECORD=1. Run it ten times. If it goes from “occasionally red and three seconds slow” to “green and instant,” you’ve found your boundary. Move the quality question into a separate scheduled job, and let your PR gate get back to testing your code instead of OpenAI’s mood.