Stop Asserting LLM Text — Test the Contract in Playwright

Your product just shipped an AI feature — a chatbot, an AI search box, a “summarize this” button — and your Playwright suite is now red on a third of its runs even though nothing is broken. The instinct is to slap { retries: 3 } on the spec and move on. That’s the wrong fix, and it’s the one that quietly hides real regressions. The right fix is to stop asserting on what the model says and start asserting on the contract the feature has to honor.

That distinction is the entire post. A deterministic test asserts an exact value because the system under test is deterministic. An LLM feature isn’t, so an exact-string assertion is testing a coincidence — it passed because the model happened to phrase it that way today. When the phrasing drifts tomorrow, your test fails on a non-bug. When a genuine defect changes the meaning while keeping the phrasing, your test passes on a real bug. You need assertions that fail on the second case and survive the first.

Why is this suddenly every SDET’s problem?

Because AI features moved from demos into the products you already test. Around 92% of Fortune 500 companies now report using generative AI in their workflows, and 67% of organizations worldwide have adopted LLMs to support operations. The number of LLM-powered applications was projected to reach 750 million globally by 2025. Translation: the non-deterministic surface area didn’t ask permission before landing in your regression suite.

I’ve watched this play out at two enterprises in the last year. The pattern is identical: a product team ships an LLM feature, QA writes E2E tests the way they always have — navigate, act, expect(locator).toHaveText('...') — and within a sprint the suite has a chronic flake that everyone learns to ignore. That’s the real cost. Not the red builds, but the moment your team stops trusting the suite and starts re-running it on autopilot. A test nobody trusts is worse than no test, because it still costs CI minutes and still gives a green check mark that hides real bugs.

Why doesn’t temperature 0 save your test suite?

Because temperature 0 only makes the token-sampling step deterministic — it does nothing about the runtime conditions underneath. The popular advice is “just set temperature: 0 and the output is fixed.” It isn’t, and treating it as if it were is how you build a suite that’s flaky in a way you can’t reproduce locally.

The research here is unambiguous. Thinking Machines’ analysis sampled 1,000 completions at temperature 0 from a single model and got 80 distinct outputs, with runs diverging as early as token 103 — purely from numerical non-determinism in batched GPU kernels, not from sampling. OpenAI is explicit that its API can only ever be “mostly deterministic” regardless of temperature, because load balancing and hardware updates shift the math under you. Academic work on the “non-determinism of deterministic LLM settings” confirms the same across providers.

So the deterministic inputs you control (prompt, retrieved context, tool schema) do not produce a deterministic output. Your assertions have to live on the side of the equation you can pin down: the structure and the constraints, not the words.

What should you actually assert on?

Assert the contract — the set of properties the output must satisfy no matter how the model phrases it. In practice that’s three layers, cheapest and most reliable first: structure, invariants, and only then meaning. The first two are deterministic and should carry 80% of your coverage. The third is where most teams start, and it’s the least reliable place to live.

Layer 1: Structure (deterministic)

Structure is free if your feature emits structured output. Force the model into a JSON schema and validate it with the same rigor you’d apply to any API contract test — types, required fields, enums, ranges. The model’s prose can vary infinitely; the shape cannot.

import { test, expect } from '@playwright/test';
import { z } from 'zod';

const SummarySchema = z.object({
  summary: z.string().min(20).max(600),
  sentiment: z.enum(['positive', 'neutral', 'negative']),
  actionItems: z.array(z.string()).max(5),
});

test('summary response honors its schema', async ({ request }) => {
  const res = await request.post('/api/summarize', { data: { docId: 'ticket-4471' } });
  const parsed = SummarySchema.safeParse(await res.json());
  expect(parsed.success, parsed.error?.toString()).toBe(true);
});

Layer 2: Invariants (deterministic)

Invariants are the deterministic facts the answer must respect regardless of wording. A summary of a 3-sentence ticket should not be longer than the ticket. A currency total in a generated response must equal the number in your fixture. A RAG answer must cite a document ID that actually exists in the retrieved set. These are exact assertions — they’re just exact about constraints, not phrasing.

Notice the durable version leans on a data-order-ids attribute the feature emits for exactly this purpose. Getting product to expose machine-readable hooks alongside generated prose is a senior move — it’s the AI-era equivalent of asking for stable data-testids, and it’s how you keep non-determinism from leaking into flaky tests.

How do you assert meaning without asserting words?

Use an LLM-as-judge — a second model that scores the output against a rubric — but treat it as a noisy gauge with a threshold, never as an oracle. Sometimes structure and invariants aren’t enough; you genuinely need to know whether the answer is relevant, grounded, and not toxic. That’s a semantic check, and a model is the only practical tool for it at E2E speed.

The catch is reliability. In expert domains, LLM judges agree with human subject-matter experts only 60–68% of the time, and even the best models display a tendency toward lenient scoring. So a judge is a probabilistic instrument: useful for catching gross failures, useless as a tight equality check. Calibrate it like one — give it a narrow rubric, force a numeric score, and assert a threshold with margin, not a perfect pass.

export async function judge(answer: string, rubric: string): Promise<number> {
  const res = await callJudgeModel({
    system: 'Score 0-10 how well the ANSWER satisfies the RUBRIC. Reply with only a number.',
    user: `RUBRIC:\n${rubric}\n\nANSWER:\n${answer}`,
    temperature: 0,
  });
  const score = Number(res.trim());
  if (Number.isNaN(score)) throw new Error(`Judge returned non-numeric: ${res}`);
  return score;
}

test('support answer is grounded and on-topic', async () => {
  const answer = await askBot('How do I reset my 2FA?');
  const score = await judge(answer, 'Answer must explain 2FA reset using only documented steps.');
  expect(score).toBeGreaterThanOrEqual(7); // threshold, not equality
});

One hard rule: never let the judge gate the build on its own opinion the way I’d never let an LLM decide flaky-vs-real instead of triaging. Log every score, alert on threshold breaches, and keep a human reviewing the borderline band. The judge widens your coverage; it does not get to be the final authority.

How do you make a non-deterministic test stable in CI?

You stop demanding that a single run be reliable and start measuring a distribution. A deterministic test runs once and means it. A probabilistic feature needs a sampling budget — run the check a few times and assert that it passes at a rate you’ve defined, the same way you’d reason about a flaky integration rather than pretending it’s binary.

For the live-sampling step, expect.poll gives you a clean way to run an assertion repeatedly without hand-rolling a loop, and Playwright’s auto-retrying assertions already understand the difference between “eventually true” and “true once.”

test('relevance holds across samples', async () => {
  let passes = 0;
  for (let i = 0; i < 5; i++) {
    const answer = await askBot('Where is my refund?');
    if ((await judge(answer, 'Must address refund status.')) >= 7) passes++;
  }
  expect(passes).toBeGreaterThanOrEqual(4); // 4-of-5, a measured bar
});

What I’d do differently

The first AI feature I tested, I lost the better part of three days chasing a ~4% flake on a chatbot spec. The assertion was toContainText('Your order has shipped'). The bug, it turned out, was that the model occasionally said “Your order is on its way” — semantically perfect, lexically different. I was debugging the test framework, the network, the worker isolation, everything except the actual problem: I had written a deterministic assertion against a non-deterministic system and called the resulting noise “flakiness.”

What I’d do now, from day one: refuse to write a single exact-string assertion against model output. Ask product for machine-readable hooks before the feature ships, not after. Push 80% of coverage into structure-and-invariant checks that are fully deterministic, reserve the judge for the semantic 20%, and measure pass-rate instead of papering over it with retries. The teams that do this have AI-feature suites they trust. The teams that don’t have a retries: 3 and a slow erosion of faith in every green build.

Your one concrete next step

This week, take your flakiest AI-feature spec and split its single assertion into the three layers. Pull out one structural check (does the response parse against a schema?), one invariant (does a number or ID match your fixture exactly?), and move the fuzzy “does it sound right” part behind a judge with an explicit threshold. You’ll usually find that two of the three are deterministic and never needed a model at all — and that the spec stops flaking the moment you stop asserting the prose.