Your 30-Case AI Eval Can't Tell If the Model Got Worse

You bumped the model behind your support assistant from one snapshot to the next, reran your 30-case golden eval, saw 28 green, and shipped. Here’s the uncomfortable part: that 28/30 is not evidence the new model is as good as the old one. It’s not evidence it’s worse, either. With a set that small, a real quality regression and pure sampling noise produce the same number — and you just promoted a model on a coin flip you mistook for a measurement.

This is the failure mode that replaced flaky selectors. Vendors reissue model snapshots on their own schedule, your prompt engineers tweak a system message, someone nudges temperature — and the only thing standing between that change and production is an eval set that was sized to “feel thorough,” not to detect anything. Comparing two models by their aggregate pass rates is the green check that hides a real bug, dressed up in the language of evaluation. Below is why the comparison fails, how big the set actually needs to be, and the paired test that squeezes real signal out of the examples you already have.

Why can’t you compare two models by their eval pass rates?

Because “29/30 vs 28/30” throws away the only information that matters: which specific examples changed. When you run the same golden set through both models, the results are paired — every example was answered by model A and model B. The right question isn’t “did the aggregate drop by a point?” It’s “how many examples did the new model break that the old one got right, versus how many did it newly fix?”

Anthropic’s own guidance on evaluation statistics is blunt about this. Their 2024 paper Adding Error Bars to Evals recommends that “when two models are being compared, conduct statistical inference on the question-level paired differences, rather than the population-level summary statistics.” Aggregate pass rates discard the pairing and inflate your variance for no reason. The examples where both models agree — the vast majority — tell you nothing about which is better; only the discordant pairs do.

Two new failures and one new fix out of thirty is a net of minus one — the aggregate you’d report. Run the paired test on those three discordant cases and the one-sided p-value is 0.50. You cannot call the model regressed. You also cannot call it safe. You have three data points of actual signal, and no amount of staring at “93%” changes that. The same non-determinism that makes a single LLM assertion flaky means a 30-case aggregate is a low-resolution instrument pointed at a small effect.

How big does your golden set actually need to be?

Bigger than you think, and it scales with the inverse square of the regression you want to catch. Detecting a large, obvious collapse is cheap; detecting the 2-to-5-point slide that actually reaches customers is expensive. Here’s what it takes to detect a drop from a 90% baseline at 80% power and a 5% significance level, comparing two models as independent samples:

A two-point regression from a 90%-reliable feature — the kind a vendor snapshot ships without a changelog — needs on the order of 3,800 examples to detect reliably if you compare the two runs independently. Thirty examples can only detect a catastrophe: something like a 25-plus-point cliff. Everything subtler slips through with a green eval, which is precisely how model drift ships to production while your mocked tests stay green.

The good news is that pairing buys back a lot of that cost. Because both models see identical inputs, the paired analysis cancels out per-example difficulty — the biggest source of variance — so you need far fewer examples than the independent-sample numbers above. The Anthropic paper reports large variance reductions from paired analysis and answer resampling. You still won’t get there on 30 cases, but a frozen set in the low hundreds, analyzed correctly, becomes a real gate instead of a decoration.

Use McNemar’s test, not two confidence intervals

The correct tool for paired binary outcomes is McNemar’s test. It ignores the examples both models agree on and asks a single question of the discordant pairs: if the two models were truly equal, how surprising is it to see this many new failures versus new fixes? For the small counts typical of a golden set, use the exact binomial form — no chi-square approximation, no minimum-cell-count caveats.

Implement the exact McNemar test

// Exact McNemar. Only discordant pairs carry signal:
// b = regressions (old passed, new failed); c = improvements (old failed, new passed).
function binomCdf(k: number, n: number, p = 0.5): number {
  let sum = 0;
  for (let i = 0; i <= k; i++) {
    let coef = 1;
    for (let j = 0; j < i; j++) coef = (coef * (n - j)) / (j + 1);
    sum += coef * p ** i * (1 - p) ** (n - i);
  }
  return sum;
}

// One-sided p: chance of >= b regressions if the models were equal.
export function mcnemarRegressionP(b: number, c: number): number {
  const n = b + c;
  if (n === 0) return 1;         // no disagreements -> no evidence of change
  return 1 - binomCdf(b - 1, n); // P(X >= b), X ~ Binomial(n, 0.5)
}

Wire it into a Playwright promotion gate

Wire it into a Playwright project so the promotion gate lives next to the rest of your suite. The eval runs both the incumbent and the candidate model over a frozen set — freeze it the same way you’d freeze AI-generated tests once they’re reviewed, because a set that drifts can’t be a baseline.

import { test, expect } from '@playwright/test';
import { mcnemarRegressionP } from './mcnemar';
import goldenSet from './golden-set.json'; // frozen inputs + pinned graders

test('candidate model is not a regression on the golden set', async ({ request }) => {
  let b = 0, c = 0; // b: regressions, c: improvements
  for (const ex of goldenSet) {
    const [oldPass, newPass] = await Promise.all([
      gradeOn(request, 'model-current', ex),
      gradeOn(request, 'model-candidate', ex),
    ]);
    if (oldPass && !newPass) b++;
    else if (!oldPass && newPass) c++;
  }
  const p = mcnemarRegressionP(b, c);
  // Fail promotion only when regressions significantly outnumber fixes.
  expect(p, `regressions=${b}, improvements=${c}, one-sided p=${p.toFixed(3)}`)
    .toBeGreaterThan(0.05);
});

What decision rule should the gate enforce?

Two rules, because significance testing has a known blind spot: on a small set it will almost never fire, and “not significant” is not the same as “safe.” Pair the statistical gate with a hard invariant so an underpowered set can’t rubber-stamp a promotion.

This gate does not belong on every pull request. Running two model versions across a few hundred graded examples is slow and it costs real money, the same way naive AI-in-CI integrations quietly run up six-figure bills. Put deterministic structure and schema checks on the PR path with a mocked model, and run the paired promotion gate only when a model, prompt, or parameter actually changes — which, given that providers change model behavior over time without warning, means on every vendor snapshot too, not just your own edits.

The one thing to do today

Stop reporting your eval as a single percentage. Change your harness to record a per-example pass/fail vector for the model currently in production and commit it next to the golden set. That one change means the next time a snapshot lands or someone rewrites a prompt, you can run a paired diff — how many examples broke, how many got fixed, and the McNemar p-value — instead of squinting at “93% vs 96%” and guessing. You will not have enough power on day one. But you’ll be measuring the right quantity, on frozen inputs, with a rule a human agreed to — and that’s the difference between a promotion gate and a comforting green number that ships regressions with confidence.