Your LLM Judge Is the Flakiest Test in Your Suite

You stopped asserting exact LLM text — good. You replaced it with an LLM-as-judge that scores whether the chatbot’s answer is “helpful and correct” — and now your suite is flaky in a way that’s even harder to debug, because the thing deciding pass/fail is itself a non-deterministic, biased model. You didn’t remove the coin flip. You added a second coin and multiplied them together.

That’s the trap I want to take apart. LLM-as-judge is the right tool for the meaning layer — there’s no regex that tells you a summary is faithful. But the moment you bolt a raw judge call into a Playwright expect, you’ve built an assertion whose verdict depends on option order, output length, and which model family wrote the answer. A senior SDET’s job isn’t to avoid the judge. It’s to turn it from a mood ring into an instrument: version-pinned, structurally constrained, bias-corrected, cached, and — this is the part everyone skips — calibrated against a human-labeled set before you trust a single green check.

Why is an LLM judge flaky when temperature is zero?

Because the judge inherits every non-determinism and bias of the model underneath it, and “temperature 0” fixes neither. The same numerical non-determinism that makes the system under test drift between runs applies to the model you’re using as a grader. But the judge adds failure modes the SUT doesn’t have: it’s sensitive to which answer you show first, how long the answer is, and whether the answer “sounds like” something the judge model would write.

These aren’t anecdotes. The foundational MT-Bench study found that strong judges like GPT-4 reach over 80% agreement with human evaluators — impressive, until you read the next number: the same GPT-4 judge was only about 65% consistent when the two answers were swapped in order, and weaker, cheaper judges flip far more often. A 2025 systematic study of position bias in LLM-as-a-judge confirms the effect is structural across pairwise and list-wise settings, not a quirk of one model.

The “Justice or Prejudice” work catalogs twelve distinct judge biases — position, verbosity, self-enhancement, and bandwagon among them. And self-preference bias means a judge systematically rewards outputs with lower perplexity, i.e. answers that look like its own writing. If your product’s answer and your judge come from the same model family, you’ve baked a thumb onto the scale and called it ground truth.

What does the naive Playwright judge get wrong?

It treats a free-text model call as if it returned a boolean. Here’s the pattern I keep finding in real suites — it looks reasonable, and it’s the source of the flake.

test('chatbot answers the refund question', async ({ page }) => {
  await page.getByRole('textbox').fill('How do I get a refund?');
  await page.getByRole('button', { name: 'Send' }).click();
  const answer = await page.getByTestId('assistant-message').innerText();

  // One ungrounded judge call, parsed by vibes
  const verdict = await judge(`Is this answer good? "${answer}"`);
  expect(verdict.toLowerCase()).toContain('yes'); // 🙈
});

Three things are broken here. The prompt asks “is this good?” — an unanchored question that invites the judge’s verbosity and self-preference biases to drive the answer. The output is free text parsed with toContain('yes'), so “Yes, but…” and “No, yet it’s not bad” both pass. And there’s no record of why it passed, so when it flips next Tuesday you have nothing to diff. This is exactly the failure mode I warned about when arguing you should let AI triage failures, not decide them — the same discipline applies when AI decides pass/fail.

How do you make the judge reproducible?

Constrain it the way you’d constrain any API contract: pin the version, force structured output against a schema, and anchor the rubric so “good” has a definition instead of a vibe. The judge should return data you can validate, not prose you have to interpret.

Force a schema’d verdict

import { z } from 'zod';

const Verdict = z.object({
  faithful: z.boolean(),   // grounded in the provided context only
  resolves: z.boolean(),   // actually answers the user's question
  score: z.number().int().min(1).max(5),
  evidence: z.string().max(280), // quote that justifies the score
});

// Pin everything that moves: model, version, temperature, seed.
const JUDGE = { model: 'gpt-4.1-2025-04-14', temperature: 0, seed: 42 };

Pin the version and anchor the rubric

Pinning the model version string matters more than people expect: providers ship silent quality updates behind floating aliases, and a judge that drifts is a test environment that drifts. The anchored rubric — faithful, resolves, a 1–5 score, and a required evidence quote — does the real work. Forcing the judge to cite the span that justifies its score suppresses verbosity bias and gives you a diffable artifact when a verdict changes.

How do you kill position and self-preference bias?

Run the judge double-blind and take a quorum. For any pairwise or reference-based comparison, evaluate both orderings and only trust a verdict that survives the swap; for single-answer scoring, sample the judge a few times and require agreement. Practitioners converge on not trusting a judge until it clears roughly 75–80% agreement with a human-labeled set — treat anything below that as an unreliable instrument.

export async function gradeFairly(answer: string, context: string) {
  // Double-blind: score in both orderings, discard order-sensitive verdicts.
  const [a, b] = await Promise.all([
    score({ first: answer, second: context, ...JUDGE }),
    score({ first: context, second: answer, ...JUDGE }),
  ]);
  if (a.faithful !== b.faithful) {
    return { stable: false, faithful: false, reason: 'order-sensitive' };
  }
  // Quorum on the numeric score: median of N samples, not a single roll.
  const scores = await sampleScores(answer, context, JUDGE, 3);
  return { stable: true, faithful: a.faithful, score: median(scores) };
}

Why order-sensitive verdicts get thrown out

A verdict that disagrees with itself when you swap argument order is, by definition, not measuring the answer — it’s measuring position. Dropping those is how you stop importing the judge’s ~35% inconsistency into your suite. To blunt self-preference bias, use a judge from a different model family than the system under test; never let a model grade its own homework.

How do you make reruns deterministic?

Cache the verdict by a hash of its inputs, so an identical answer and rubric never gets re-judged. This is the single highest-leverage move for CI stability: a retried job, a rerun on a flake, or a no-op rebuild should return the same verdict from cache, not a fresh dice roll. It also slashes the token bill — relevant if you remember that naive AI integrations quietly cost six figures a year in CI.

import { createHash } from 'node:crypto';

function verdictKey(answer: string, context: string, rubricVersion: string) {
  return createHash('sha256')
    .update(`${rubricVersion}\n${context}\n${answer}`)
    .digest('hex');
}
// Cache hit => identical verdict on every rerun. Cache miss => grade once, store.

How do you know the judge itself is correct?

You write a test for the judge. Keep a small, version-controlled calibration set of answers you’ve labeled by hand — clearly good, clearly bad, and a few deliberate near-misses — and run the judge against it in CI. If the judge’s agreement with your labels drops below threshold, you fail the judge, not the product. This is the step that separates a measured instrument from a guess.

test('judge agrees with human labels', async () => {
  const labeled = loadGoldenSet(); // committed, reviewed in PRs
  const results = await Promise.all(labeled.map(g => gradeFairly(g.answer, g.context)));
  const agreement = results.filter((r, i) => r.faithful === labeled[i].faithful).length / labeled.length;
  expect(agreement).toBeGreaterThanOrEqual(0.8); // fail the JUDGE, not the product
});

Keep the judge out of the hot path

One architectural note that saves teams from themselves: don’t put the judge inline in your blocking E2E suite. Structure and invariant checks — schema, required fields, no-hallucinated-links — are deterministic and belong in the fast suite that gates merges. Run the LLM-judge meaning checks as a separate graded job with its own threshold and its own dashboard, the same way you’d treat a green check that might be hiding a real bug with suspicion. A probabilistic grader gating every PR turns your merge queue into a slot machine; a probabilistic grader tracked as a quality metric tells you something true over time.

The one thing to do today

Take your flakiest AI-feature test, and add one assertion that has nothing to do with the judge: run the judge twice on the same answer and assert the two verdicts match. If they don’t, you’ve just proven the test was never measuring your product — and you now have the reproduction case to fix it with the rubric, swap, and cache pattern above. Pin the judge, calibrate it against a human-labeled set, and move the meaning checks out of your merge gate. A judge you’ve measured is an asset. A judge you haven’t is just a more expensive flake.