Your AI Test Should Be Allowed to Fail — By Design

Your AI-feature test asserts the chatbot answered the refund question correctly, and it went green. Run it again — same input, same build — and it’s red. Nothing broke. You’re asking a probabilistic system a yes/no question and treating one coin flip as a verdict. The honest test isn’t “did it pass?” It’s “does it pass often enough?” — and that’s a number with a confidence interval, not a boolean.

This is the part most teams get backwards. They feel the flake, reach for Playwright’s retries, watch the suite go green, and ship. What they’ve actually done is convert a reliability regression into a silent pass, which is far more dangerous than a flake you can see. The senior move is to stop pretending an AI assertion is deterministic and start gating on pass@k with an explicit reliability budget — the same discipline you’d bring to an SLO. Below is the math, the Playwright wiring, and the one config change that keeps retries from lying to you.

Why does a correct AI test fail half the time?

Because the feature under test has a true success rate below 100%, and a single assertion samples it exactly once. If your checkout assistant answers correctly 90% of the time, a one-shot expect is a Bernoulli trial that fails 1 run in 10 — indistinguishable from a real bug at the moment it goes red. The same model drift that makes LLM features non-deterministic between runs means there is no single “right answer” to snapshot against.

The evaluation literature already solved the measurement problem. pass@k — introduced with OpenAI’s Codex/HumanEval work — is the probability that at least one of k independent samples succeeds. It exists precisely because LLM-based systems exhibit run-to-run variability that a single sample can’t characterize. The mistake is importing the metric without importing the statistics behind it.

Why is retries the wrong fix for AI flake?

Because retries answer “did any attempt pass?” when the question you actually care about is “how often does it pass?” — and those diverge exactly when quality regresses. Playwright’s retry mechanism reruns a failed test up to N times and marks it passed if any attempt succeeds. That’s pass@(retries+1), and it’s great for genuine infrastructure flake: a slow network, a cold container, a race you haven’t fixed yet.

It’s poison for model quality. Watch what retries do to a regression:

If your feature’s true success rate silently falls from 90% to 60%, retries: 2 computes 1 − (1 − 0.6)³ = 93.6% — your suite still passes almost every time. You’ve built a green check that hides a real bug, and you’ll find out from a customer. Retries suppress the signal you most need to see.

How many runs before an AI test is actually “green”?

Enough runs that the lower bound of your estimated success rate clears your target — not enough that one lucky pass squeaks through. You’re not asserting a boolean anymore; you’re estimating a proportion and gating on its confidence interval. Sample the feature N times, count the successes, and compute a Wilson lower bound on the true success rate.

// Wilson score lower bound — stable at small N, unlike the naive normal interval.
export function wilsonLower(successes: number, n: number, z = 1.96): number {
  if (n === 0) return 0;
  const p = successes / n;
  const denom = 1 + (z * z) / n;
  const centre = p + (z * z) / (2 * n);
  const margin = z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n));
  return (centre - margin) / denom;
}

Why 18/20 isn’t a 90% feature

Here’s the number that makes the point concrete. Observe 18 passes out of 20 — a healthy-looking 90% — and the 95% Wilson lower bound is only ≈0.70. You cannot honestly claim a 95%-reliable feature from 18/20. The interval is telling you that you don’t have the evidence yet, which is the entire job of the gate.

The rule of three for zero-failure runs

For the clean case — zero failures — lean on the rule of three: if you see 0 failures in N runs, you can be ~95% confident the true failure rate is below 3/N. So a gate of “0 failures in 60 samples” buys you ~95% confidence the feature fails less than 5% of the time. That single heuristic is worth more than any retry count you’ll ever tune.

import { test, expect } from '@playwright/test';
import { wilsonLower } from './reliability';

test('checkout assistant clears the reliability budget', async ({ request }) => {
  const N = 60, target = 0.95;
  let passes = 0;
  for (let i = 0; i < N; i++) passes += (await answersRefundQ(request)) ? 1 : 0;

  const lower = wilsonLower(passes, N);
  // Gate on the LOWER bound, not the point estimate — that's what kills false greens.
  expect(lower, `pass=${passes}/${N}, 95% lower=${lower.toFixed(3)}`)
    .toBeGreaterThanOrEqual(target);
});

Where should the reliability gate run?

Not on every pull request — it’s too slow and too expensive for the merge path. Sixty live model calls per feature, times a dozen AI features, is real latency and a real bill; recall that naive AI-in-CI integrations quietly cost six figures a year. Split the suite by determinism and put each half where it belongs.

Isolate the gate in its own Playwright project

That last point deserves its own config, because it’s the one line teams forget. Give the reliability gate a dedicated project so the default retry policy can’t contaminate its statistics:

export default defineConfig({
  retries: 2, // fine for the deterministic UI suite — infra flake only
  projects: [
    { name: 'ui', testMatch: /.*\.spec\.ts/ },
    {
      name: 'ai-reliability',
      testMatch: /reliability\.spec\.ts/,
      retries: 0, // every run is a sample — retrying would inflate the pass rate
    },
  ],
});

What number do you actually gate on?

Write it down as a service level objective before you argue about the threshold, because the threshold is a product decision, not a testing one. “The checkout assistant answers correctly on ≥95% of runs, measured over a rolling 200 samples” is a sentence a PM can sign off on and an on-call engineer can be paged against. Once it’s a documented budget, pass@k stops being a clever trick and becomes an accountability line.

Be honest that pass@k is a blunt instrument — it collapses a distribution into one number, and a 2025 Bayesian critique argues you should model per-sample success probability directly for tighter estimates. For a merge gate, the Wilson bound is the right amount of rigor; save beta-binomial modeling for when you’re comparing two model versions and need to detect a 2-point difference. And whatever grades “correct” — a schema check or a reproducible LLM judge — has to be pinned and calibrated first, or you’re computing a confidence interval on noise.

The one thing to do today

Take your single flakiest AI-feature test and wrap its assertion in a 30-run loop that counts passes and prints the ratio. Don’t gate on it yet — just look at the number. If it reads 22/30, you’ve been shipping a 73%-reliable feature and calling it green, and you now have the evidence to set a real budget. Move the deterministic checks to your PR gate with a mocked model, stand up the pass@k loop as a nightly graded job with retries: 0, and give it an SLO a human agreed to. An AI test that’s allowed to fail — within a measured budget — tells you the truth. One that always passes is just lying more confidently.