Your AI Test Failed. Prove Which Prompt Did It.

The nightly eval fails: expected the summary to mention the refund, received a generic paragraph. You rerun it locally and it passes. You have the screenshot, the DOM snapshot, and a Playwright trace showing a request went out and a response came back. What you do not have is the system prompt that was actually assembled, the retrieved chunks that went into it, the model the provider routed you to, or the token count. The failure isn’t unreproducible because the model is random. It’s unreproducible because your test threw away the only evidence that mattered.

For an AI feature, the DOM is the weakest oracle in your system. Everything that determines whether the output is correct — prompt version, retrieved context, tool schema, model checkpoint, temperature, token budget — lives server-side and never crosses into your assertion. The fix is to stop treating the browser as the only place you can observe. Propagate a W3C trace context from every Playwright test into the backend, let OpenTelemetry’s GenAI conventions capture the LLM call as a span, and then assert on that span. The trace becomes both your debugger and your oracle.

Why can’t you reproduce an AI test failure?

Because a failed assertion tells you the output and hides the input. In a classic web test, the input is the fixture data you seeded — you own it, so a rerun reproduces it. In an AI feature, the input is assembled at request time out of things your test never touched: a prompt template that a product manager edited last Tuesday, a RAG retrieval that returned different chunks because the index reindexed overnight, and a model version the provider swapped under the same string.

That gap is expensive well before you add AI to the mix. Roughly 26% of developer time goes to reproducing and fixing failing tests, and one 2026 survey found 34% of DevOps engineers spend over 20 hours a week debugging pipeline failures they can’t reproduce locally. AI features take that number and multiply it, because the non-reproducibility is now a property of the system rather than a property of your environment.

I have watched a senior engineer spend two days on an eval failure that turned out to be a retrieval bug — the vector store had been swapped to a new embedding model, so the “refund policy” chunk stopped ranking. Nothing in the test output said “retrieval.” A trace would have said it in four seconds.

What should a failing AI test actually hand you?

One clickable trace ID. Not logs, not a screenshot — a link that opens the exact server-side execution the test triggered: the HTTP span, the retrieval span, the LLM span with its model and token counts, and the tool calls the agent made. Everything else is archaeology.

Getting there is three moving parts: mint a span per test, inject the W3C traceparent header so the backend joins your trace, and attach the resulting trace URL to the Playwright report. Then, once the trace exists, you get a second and much better payoff — you can assert on it.

How do you join a Playwright test to a backend trace?

Give the test its own span and push the trace context into every request the browser makes. Playwright fixtures make this clean: one fixture wraps the test in an OTel span and sets the traceparent header on the browser context, so both the page navigation and any fetch from your app carry the same trace ID the test owns.

The fixture that carries the trace ID

Here’s the whole thing. It’s short because OTel does the propagation for you:

import { test as base } from '@playwright/test';
import { trace, context, propagation, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('e2e');

export const test = base.extend<{ traceId: string }>({
  traceId: async ({ context: browserContext }, use, testInfo) => {
    const span = tracer.startSpan(testInfo.title);
    const ctx = trace.setSpan(context.active(), span);

    // Serialize the active context into W3C headers and hand them to the browser.
    // Every navigation and fetch from the page now carries the test's trace ID.
    const headers: Record<string, string> = {};
    propagation.inject(ctx, headers);
    await browserContext.setExtraHTTPHeaders(headers);

    const traceId = span.spanContext().traceId;
    await use(traceId);

    if (testInfo.status !== testInfo.expectedStatus) {
      span.setStatus({ code: SpanStatusCode.ERROR });
      await testInfo.attach('trace', {
        body: `https://your-otel-ui/trace/${traceId}`,
        contentType: 'text/plain',
      });
    }
    span.end();
  },
});

What does the LLM span give you that the DOM doesn’t?

It gives you the input. OpenTelemetry’s GenAI semantic conventions standardize the attributes every provider SDK now emits: gen_ai.system, gen_ai.operation.name, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and tool-call attributes for agents. Vendors have converged on it — Datadog maps the conventions natively rather than asking you to instrument twice.

Three assertions become possible the moment those spans exist, and each one catches a bug class the DOM structurally cannot see:

import { test } from '../fixtures/trace';
import { expect } from '@playwright/test';
import { fetchSpans } from '../helpers/otel';

test('refund summary calls the pinned model exactly once', async ({ page, traceId }) => {
  await page.goto('/tickets/8842');
  await expect(page.getByTestId('summary')).toBeVisible();

  const spans = await fetchSpans(traceId, { name: 'gen_ai.chat' });

  // 1. Double-invoke: a React effect firing twice bills you twice. The DOM looks fine.
  expect(spans).toHaveLength(1);

  // 2. Silent routing: you pinned a snapshot, the gateway served the alias.
  expect(spans[0].attributes['gen_ai.request.model']).toBe('claude-sonnet-4-5-20250929');

  // 3. Context bloat: a retrieval regression that quietly triples your prompt.
  expect(spans[0].attributes['gen_ai.usage.input_tokens']).toBeLessThan(4000);
});

The second assertion is the one that pays rent. A summary rendered by a model you didn’t pin still renders — it fills the div, it reads plausibly, and your DOM assertion passes. You only find out you’ve been silently routed to a different checkpoint when quality slides three weeks later. This is the same silent-swap failure I wrote about in mocked AI tests and model drift, except here you catch it in the run that caused it rather than in a nightly digest.

The third assertion is a cost regression test in disguise, and it composes with the token budgets you should already be putting on your AI agents. A retrieval that starts returning ten chunks instead of three doesn’t fail any assertion you own today. It just triples your bill.

Doesn’t this make my tests depend on my observability stack?

Partly, and you should bound that dependency deliberately. The rule I’d apply: trace injection is always on; span assertions live in one lane.

Injecting traceparent costs nothing and cannot fail a test — it’s a header. Turn it on for the whole suite, including the mocked lane, because the debugging payoff is free. Span assertions, on the other hand, require a live collector, so they belong in the same non-mocked lane where you run your real-model evals — the second lane in a two-lane mock-in-CI, eval-nightly setup. Locally, swap the exporter for an in-memory one and assert against it with zero infrastructure.

Two more things will bite you, and both are boring.

Tail sampling will eat your trace

Most production collectors sample. If your test’s trace gets dropped, your span query returns an empty result and you will spend the afternoon blaming the fixture. Force the sampled flag (01) on the test’s traceparent, and configure the collector to always keep traces carrying your test’s tracestate marker. A trace you can’t retrieve is a trace you didn’t emit.

Spans export asynchronously

The test finishes, the process exits, the batch exporter never flushes, and your assertion queries a span that is still sitting in a buffer. Call forceFlush() on the tracer provider in a teardown fixture, and back the span query with a short poll — a few hundred milliseconds of retry beats a heisen-failure that only reproduces on the CI runner.

What about the prompt text itself?

Capture it, but treat it like a hazardous material. The GenAI conventions are explicit that prompt and completion content capture is opt-in, and the spec is still marked Development — attribute names have moved, and you should pin your instrumentation and set OTEL_SEMCONV_STABILITY_OPT_IN deliberately rather than letting an SDK upgrade silently rename gen_ai.usage.prompt_tokens under you.

The bigger issue is what’s inside the prompt. A trace that contains the full assembled context contains whatever your RAG layer retrieved — which, in a support console, is customer PII. Redact before export, not after. I covered the boundary rules in redacting Playwright traces before they reach an LLM, and the same rules apply to the span content: scrub server-side, in the exporter, on the way out.

And don’t assert on the prompt string. Assert on its properties — length, token count, that the retrieved chunk IDs include the one you seeded, that the system prompt version attribute matches the version you pinned. String-matching a prompt template is a test that breaks the first time someone fixes a typo.

What I’d do differently

I’d have wired the trace ID into the Playwright HTML report on day one, before writing a single span assertion. We built the assertions first because they felt like the “real” engineering, and spent a month with a nightly eval suite that still forced people to grep the collector by timestamp to find their own failure. The attachment is twenty lines. It changed the mean time to diagnosis more than anything else in this post.

I’d also have used the trace as a triage input earlier. Once the trace ID is on the failure, the natural next move is to hand the span — model, tokens, tool calls, retrieved chunks — to an LLM and let it write the first-pass diagnosis, the pattern I described in letting AI triage your Playwright failures. A triage agent reading a full trace is dramatically better than one reading a stack trace, for exactly the reason this whole post exists: it can finally see the input.

Your one concrete next step: add the traceparent fixture to your AI feature’s spec file — with the sampled flag forced on — and attach the trace URL on failure. Don’t add a single span assertion yet. Just make the next failure clickable, and see how long the following bug takes to diagnose compared to the last one.