Mocked AI Tests Pass. That's How Model Drift Ships.

Your AI feature has a summarizer, a suggest-a-reply box, or a support chatbot. Its Playwright suite is green, and it’s green because you mocked the model — you stubbed the provider response so the test is deterministic and fast. That mock is correct engineering. It’s also the reason you will find out about a quality regression from a customer instead of from CI. The provider can swap the model behind the same version string, your mock will never notice, and your suite will keep reporting a feature that no longer works the way it did last week.

Here’s the whole argument: mocking the LLM is the right call for your blocking CI gate, but it can only ever test your code — the plumbing, the parsing, the error states. It cannot test the thing your feature actually depends on, which is the model’s behavior. You need a second, separate suite that hits the real model on a schedule, scores the output semantically, and alerts — not blocks — when quality drifts. This post is how to build that second gate with Playwright.

Why do mocked AI tests miss real regressions?

Because a mock freezes the one variable your feature can’t control. When you stub the provider, every assertion downstream is really an assertion about your parsing and rendering code against a response you wrote in 2026. The model that ships that response in production is a moving target, and your mock is a photograph of it.

This isn’t hypothetical. The most-cited measurement of it is the Stanford/Berkeley study by Chen, Zaharia, and Zou, which evaluated the same GPT-4 endpoint three months apart: its accuracy at identifying prime numbers fell from 84% in March 2023 to 51% in June 2023 — same API, same version string, a 33-point drop in a single capability. The paper’s own conclusion was that teams need to “continuously monitor” model behavior, because point-in-time validation tells you nothing about next month.

And it’s worse than “the model changed,” because the model isn’t even stable within a single day. Setting temperature: 0 does not buy you determinism: as Thinking Machines Lab demonstrated in 2025, the real culprit is batch-size-dependent reduction kernels on the inference server — the same prompt dispatched under a different dynamic batch size runs its floating-point reductions in a different order and can return a different token. So your feature’s real backend is non-deterministic and silently versioned. A string-equality assertion against that is a test that will lie to you.

What should a drift gate actually check?

It should answer one question your mocked suite can’t: does the live model still produce output good enough for this feature? Not “is the output byte-identical to a golden string” — that assertion is guaranteed to flake against a real model. The gate checks whether the output still means the right thing and still satisfies the structural contract your UI depends on.

That split isn’t cosmetic — flakiness is already the dominant tax on test suites, with the share of teams reporting flaky tests climbing from 10% in 2022 to 26% in 2025 across millions of analyzed builds. Force a real model into an exact-match assertion and you’re volunteering to grow that number. The split keeps the fuzzy part scored and the strict part strict:

The mistake I see teams make is trying to force the second kind into the first — writing expect(summary).toContain("refund") and then chasing the flake for a month. Don’t. Score it.

How do you assert on non-deterministic output without flaking?

Use a graded assertion with a threshold, not an exact match. Two techniques carry the load: embedding similarity for “is this close to a known-good answer,” and an LLM-as-judge rubric for “is this actually good.” Industry eval guides converge on running both — semantic similarity against reference answers catches meaning-level failures cheaply, and an LLM judge catches the quality regressions before they reach users.

Assert the structural contract first

Start with the deterministic contract, because it’s the part that should block:

import { z } from 'zod';

// The shape your UI actually renders — assert it exactly, always.
export const SummarySchema = z.object({
  summary: z.string().max(280),
  citations: z.array(z.string()).min(1),
  // A silent model swap loves to drop this field entirely
  confidence: z.number().min(0).max(1),
});

export function assertContract(raw: string) {
  return SummarySchema.parse(JSON.parse(raw)); // throws on drift in shape
}

Then grade the meaning, don’t match it

Then the graded layer. Keep the golden set small and representative — the queries that have burned you before, not a thousand random ones. A curated golden subset your team trusts is worth more than volume; it’s the subset used specifically for gating and regression detection.

import { cosineSimilarity, embed } from './embeddings';

// Fuzzy match against a known-good reference, not the raw string.
export async function semanticScore(actual: string, reference: string) {
  const [a, b] = await Promise.all([embed(actual), embed(reference)]);
  return cosineSimilarity(a, b); // 0..1
}

export async function judgeFaithfulness(source: string, summary: string) {
  // Rubric prompt: score 1-5, MUST cite a span from source for any claim.
  // Validate the judge against human labels before you trust it.
  const verdict = await runJudge({ source, summary });
  return verdict.score; // 1..5
}

Now the Playwright test reads like a threshold check, and thresholds are what make it survive real non-determinism:

import { test, expect } from '@playwright/test';
import { golden } from './golden-set';
import { assertContract, semanticScore, judgeFaithfulness } from './score';

for (const item of golden) {
  test(`drift: ${item.id}`, async ({ request }) => {
    const raw = await callLiveFeature(request, item.input);
    const parsed = assertContract(raw);            // hard fail on shape
    const sim = await semanticScore(parsed.summary, item.reference);
    const faith = await judgeFaithfulness(item.source, parsed.summary);
    expect(sim, 'semantic drift from reference').toBeGreaterThan(0.82);
    expect(faith, 'faithfulness regression').toBeGreaterThanOrEqual(4);
  });
}

The thresholds (0.82, 4) are not magic numbers — you set them by running the gate against the current model for a week and taking the floor of the observed range, minus a margin. That baseline is your regression detector.

Where does this run — and why not in the blocking pipeline?

It runs on a schedule, in its own Playwright project, and it alerts instead of blocking. This is the part people get wrong: they wire the real-model eval into the PR gate, it flakes because the model is non-deterministic, everyone adds retry: 3, and within a month the gate is ignored. A drift check that blocks merges will be disabled by your own team. A drift check that pings Slack at 6 a.m. with “faithfulness dropped from 4.6 to 3.1 overnight” gets read.

You don’t need a new tool for the wiring — Playwright’s custom reporter API already gives you structured pass/fail and per-test scores you can post to a dashboard or Slack. Isolate it as a separate project so it never touches PR runs:

export default defineConfig({
  projects: [
    { name: 'e2e', testDir: 'tests/e2e' },        // mocked, blocks PRs
    {
      name: 'drift',
      testDir: 'tests/drift',                       // real model, scheduled
      retries: 0,                                    // NEVER retry a drift eval
      fullyParallel: false,                          // rate limits are real
    },
  ],
});

Then schedule it, cap the spend, and route the result to a human:

on:
  schedule:
    - cron: '0 6 * * *'   # 6 a.m. daily, off the PR path
  workflow_dispatch: {}
jobs:
  drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx playwright test --project=drift --reporter=json
      # Post a digest to Slack; do NOT fail the branch protection

What this caught in production

At a fintech I worked with, the “explain this transaction” feature summarized card charges for a support console. The mocked suite was green for eleven straight weeks. Then the daily drift job flagged a faithfulness drop overnight — the live model had started appending a cheerful “Let me know if you have questions!” to every summary, which blew past the 280-character contract on long transactions and, worse, occasionally invented a merchant category the source data never contained. No code had changed on our side. The provider had rolled a new checkpoint under the same model name.

Because the gate alerted instead of blocking, nobody’s Friday deploy was held hostage. We pinned to the dated model snapshot, tightened the faithfulness rubric, and shipped a prompt fix by Monday — before it reached the support team’s queue. That’s the entire value proposition: the drift gate converts a customer-reported incident into a 6 a.m. Slack message. This is the same failure mode I unpacked in the enterprise AI version gap, just caught on the way in instead of after the fact.

What I’d do differently

I’d build the golden set from real traces on day one instead of writing synthetic examples. The synthetic golden set we started with was too clean — it never included the messy, multi-currency, partially-refunded transactions that actually broke. Once we seeded the golden set from production traces (redacted — see testing LLM features in Playwright for how we scrub PII before it leaves the boundary), the gate started catching drift that mattered instead of drift on inputs no user ever sends.

Your one concrete next step: pick your single most important AI feature, curate ten golden inputs from real traffic, and stand up the deterministic contract check only. Not the semantic layer, not the judge — just the schema assertion running daily against the real model. That alone will catch the class of drift where the provider quietly drops a field or blows a length limit, and you’ll have the skeleton to add semantic scoring next week.