Temperature 0 Isn't Deterministic — Your AI Test Is Flaky

You pinned the model snapshot. You set temperature: 0. You seeded the fixture, froze the clock, and passed a seed parameter for good measure. The test still fails about once in forty CI runs, always on a different assertion, never on your laptop. So you add a retry, and the retry passes, and you move on with a small unpleasant feeling that you’ve just papered over something.

You have. Temperature 0 does not make an LLM deterministic. It makes sampling deterministic — the model picks the highest-probability token instead of drawing from the distribution. But the probabilities themselves are computed by kernels that are not batch-invariant, and the batch is composed of whatever other requests happened to be in flight when yours arrived. Your test’s output depends on a hidden input you don’t control and can’t see: the load on the inference server at the moment you called it.

Why does temperature 0 still produce different outputs?

Because greedy decoding only guarantees you take the top token — it doesn’t guarantee the top token is the same one twice. When two candidate tokens are nearly tied, a numerical difference of one part in a million in the logits is enough to flip the argmax, and from there the completion diverges permanently.

Where does that numerical difference come from? Not from GPU concurrency, which is the folk explanation everyone repeats. The forward pass of an LLM contains essentially no atomic adds, so it is run-to-run deterministic: same inputs, same bits out. The real culprit, as Horace He and Thinking Machines Lab demonstrated in September 2025, is that the reduction kernels — RMSNorm, matmul, attention — change their reduction strategy depending on batch size. Floating-point addition isn’t associative, so a different reduction order gives a different result. Your request’s numerics therefore depend on how many other people’s requests were batched alongside it.

They ran the same prompt through Qwen3-235B at temperature 0, one thousand times:

Read that middle number again, because it’s the one that explains your flake pattern. The completions were byte-identical for 102 tokens and then split. That’s why the failure looks random and localized: most of the time the divergence lands somewhere harmless, and once in a while it lands on the one clause your assertion happens to read.

Why does it only flake in CI and never on my laptop?

Because your laptop sends one request at a time and CI sends twelve. When you run the spec locally, your request usually lands in a batch of one — the same batch, every time, so the numerics are stable and the output is stable. Your Playwright workers, meanwhile, fire concurrently into a shared endpoint that is also serving three other teams and a nightly ETL job. Batch composition changes on every run, and so do the logits.

Every hour you spend making your suite faster through parallelism makes this worse, which is a genuinely annoying trade. Sharding your AI specs across twelve workers doesn’t just add contention — it actively randomizes the numerics of every request. This is a cousin of the prompt cache race that hits parallel AI tests: in both cases, the thing that changed isn’t your code, it’s the state of a shared server that your fixtures never modeled. Prefix caching and chunked prefill add another axis — SGLang’s own FAQ has long acknowledged that results aren’t deterministic at temperature 0 because of dynamic batching and the radix cache.

Measure it before you argue about it

Before you argue about this with anyone, measure it. Fifteen lines, run it against the endpoint your CI actually hits:

import { setTimeout as sleep } from 'node:timers/promises';

const ask = () => callModel({ prompt: GOLDEN_PROMPT, temperature: 0, seed: 42 });

async function probe(concurrency: number, runs = 200) {
  const outputs = new Set<string>();
  for (let i = 0; i < runs; i += concurrency) {
    // Fire `concurrency` requests at once — this is the variable that matters,
    // not temperature, not seed. It changes the batch your tokens are computed in.
    const batch = await Promise.all(Array.from({ length: concurrency }, ask));
    batch.forEach((o) => outputs.add(o));
    await sleep(50);
  }
  console.log(`concurrency=${concurrency} → ${outputs.size} unique completions / ${runs}`);
}

await probe(1);   // your laptop
await probe(12);  // your CI

On the claims-summarizer we ran at a large Canadian insurer, the numbers came back concurrency=1 → 1 unique and concurrency=12 → 9 unique. We had spent the better part of three weeks blaming the retrieval layer. The probe took an afternoon and settled it.

What should you assert instead?

Stop asserting on the text and start asserting on the properties the text must have. A completion that says “Queens, New York” and one that says “New York City” are the same answer for every business purpose you have; only your toBe() disagrees. Here’s the ladder I use, from the assertions that survive batch nondeterminism to the ones that don’t.

The diff that took our quarantine list to zero

The rewrite is smaller than you’d expect. This is roughly the diff we shipped:

test('summarizes the claim with the refund amount', async ({ page }) => {
  await page.goto('/claims/CLM-88421');
  const summary = page.getByTestId('summary');
  await expect(summary).toBeVisible();

- // Batch-size nondeterminism will flip a token here roughly once in 40 runs.
- await expect(summary).toHaveText(GOLDEN_SUMMARY);
- expect(sha256(await summary.innerText())).toBe(GOLDEN_HASH);

+ // Invariant: the number came from our fixture. If it's wrong, it's a real bug.
+ await expect(summary).toContainText('$1,240.00');
+ await expect(summary).toContainText('CLM-88421');
+
+ // Structural: the model must cite the chunk we seeded, exactly once.
+ const citations = await page.getByTestId('citation').all();
+ expect(citations).toHaveLength(1);
+ await expect(citations[0]).toHaveAttribute('data-chunk-id', 'policy-refund-v3');
});

Two things to notice. First, the new assertions are stronger, not weaker — a golden-text match would happily pass on a summary that quotes the right sentence for the wrong claim, whereas the invariant check won’t. Second, semantic quality has left this spec entirely. It didn’t get deleted; it moved to a pass@k eval gate where non-determinism is a measured property rather than a surprise. If your summary needs to be good, not just correct, that’s a rate you track, and the grounding assertions I use for RAG answers are where the real oracle lives.

Can you just buy determinism?

If you self-host, yes — and it’s real. Thinking Machines released batch-invariant kernels, and SGLang shipped them behind a flag in v0.5.3. Turn it on and 1,000 identical requests produce 1,000 identical completions, across varying batch sizes, with prefix caching and chunked prefill still enabled:

python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --attention-backend fa3 \
    --enable-deterministic-inference   # 1000 requests → 1 unique output

The bill comes as throughput. SGLang measured an average 34% slowdown with the FlashInfer and FA3 backends (down from the ~61% the original batch-invariant implementation cost), and their own recommendation is to use it for debugging and reproducibility rather than production serving. Their team’s guidance is worth quoting plainly: deterministic inference is a tool, not a default.

So my rule: run one deterministic golden-set lane; assert on properties everywhere else. A nightly lane against a batch-invariant server gives you a bitwise-stable baseline you can actually diff when quality moves — that’s the lane where model drift becomes detectable instead of merely suspected. Your PR gate keeps running against the shared endpoint at full parallelism with structural and invariant assertions, because that’s what your users actually hit.

And if you’re on a hosted API? You can’t buy it. You don’t own the batch. OpenAI’s seed parameter is explicitly best-effort, not guaranteed — the docs tell you to watch system_fingerprint for backend changes and acknowledge outputs may still differ when everything matches. seed is necessary but not sufficient, and any enterprise test strategy that assumes otherwise is building on a promise the vendor never made.

What about mocked tests — are those safe?

Yes, and this is the one place exact-match assertions are legitimate. A recorded response is a fixture; it has no logits and no batch. Assert on it byte-for-byte, because if it changes, your code changed.

That’s precisely why the two-lane setup — mocks in CI, real models nightly — holds up so well here. The mocked lane gets to be strict because nothing about it is probabilistic. The real-model lane gets to be property-based because everything about it is. The failure mode teams fall into is running one lane with mocked-lane assertions against a real endpoint, and then wondering why their pass rate is 97.5%.

What I’d do differently

I’d have run the determinism probe on day one instead of week three. We had a nightly suite with six quarantined “flaky retrieval” tests, a Jira epic to “improve embedding stability,” and a growing sense that AI features were just inherently unreliable. All six failures were tied-logit coin flips. Rewriting the assertions took two days and took the quarantine list to zero; the retrieval layer was never broken.

I’d also have separated the two questions we kept conflating. Is the output correct? is a structural question with a deterministic answer. Is the output good? is a statistical question with a distributional answer. We spent three weeks trying to answer the second question with a tool built for the first, which is a very expensive way to learn what an assertion is for.

Your one concrete next step: run the probe above against your own endpoint, at concurrency 1 and at your CI’s worker count. If the second number is greater than 1 — and it will be — then every exact-match assertion in your AI specs is a coin flip you’ve been calling heads. Go find them before your next quarantine review does.