Your Playwright Traces Leak PII to the LLM

Your CI run goes red, a reporter scoops up the Playwright trace, and a tidy little script forwards it to an LLM so the model can tell you what broke. Useful — until you remember what’s actually inside that trace.zip: the full response body of /api/customer, a real person’s email and SIN from your prod-mirrored staging data, a JWT in localStorage, and a screenshot of a filled-in checkout form. You didn’t send the model a stack trace. You sent it regulated customer data, over the wire, to a third party that retains it by default. That’s not a triage step. That’s an unlogged data egress.

I’ve now seen three teams bolt AI failure-triage onto a Playwright suite without a single line of redaction in between, and every one of them would have failed a privacy audit on the spot. The fix isn’t to abandon AI triage — letting AI triage failures instead of deciding them is the highest-leverage use of a model in a test pipeline right now. The fix is a redaction gate: a deterministic step that sits between your trace artifacts and any LLM call, sends the shape of a failure instead of its contents, and is itself covered by a test. This is the part everyone skips, and it’s the part a senior SDET owns.

Why now? Because AI triage is being wired straight into CI

Because the convenient pattern and the compliant pattern look identical in a demo and opposite in an audit. Teams are racing to put models in the pipeline, and the data governance is lagging the adoption by a wide margin. IBM’s 2025 Cost of a Data Breach Report puts the global average breach at $4.44 million, and found that organizations with high “shadow AI” — unsanctioned tools quietly handling company data — paid an extra $670,000 per breach, with those incidents disproportionately exposing customer PII.

The governance gap is the real story. In the same report, 97% of organizations that had an AI-related security incident lacked proper AI access controls, and 63% had no AI governance policy at all. A CI job that POSTs trace contents to an API endpoint nobody reviewed is shadow AI wearing a DevOps badge.

What is actually inside a Playwright trace?

Far more than the error you wanted to send. A trace.zip is a complete forensic recording of the test, and per the Playwright trace viewer docs it bundles DOM snapshots for every action, full network request and response bodies, console logs, and a screenshot film strip. The tracing API captures all of this by default once you enable screenshots and snapshots.

Walk through what that means for a single failed checkout test:

The response body for /api/customer is the whole record, not the one field your assertion touched. The DOM snapshot freezes the form with the values in it. The storage state holds live credentials. And the screenshots render all of it as images, which your regex-based scrubber can’t read at all. PII hiding in operational data isn’t a fringe risk either — an empirical study of sensitive information in logs found it pervasive across real-world systems, and broader analyses show 90% of mobile data leaks expose a username and 85% a password. Traces are denser than logs. They carry more.

Doesn’t the LLM provider just delete it?

Not by default, and “default” is the setting your CI runs under. OpenAI’s data controls documentation states that standard API inputs and outputs are retained for up to 30 days for abuse monitoring. Zero Data Retention exists, but it’s an enterprise feature that isn’t self-serve — your account team has to enable it on eligible endpoints. Training exclusion is yet a separate control. So unless someone explicitly negotiated and configured all of this, your customer’s SIN is sitting in a third party’s logs for a month, governed by a contract you probably never signed for this use case.

This is the same class of mistake as trusting an AI agent with capabilities you didn’t scope — I made the parallel argument about how your app under test can hijack a Playwright MCP agent. Inbound or outbound, the discipline is identical: never let the convenient path move data you haven’t deliberately decided to move.

How does the leak get written? (the trap)

It looks responsible. A reporter fires only on failure, gathers “context,” and sends it for triage. The problem is what counts as context.

class TriageReporter implements Reporter {
  async onTestEnd(test: TestCase, result: TestResult) {
    if (result.status !== 'failed') return;

    const payload = {
      error: result.error?.message,
      stdout: result.stdout.join('\n'),
      // The response body for /api/customer — full JSON, PII and all
      bodies: result.attachments
        .filter(a => a.name === 'response-body')
        .map(a => a.body?.toString('utf8')),
      screenshot: result.attachments.find(a => a.name === 'screenshot')?.body,
    };
    await fetch(process.env.LLM_ENDPOINT!, {
      method: 'POST',
      body: JSON.stringify(payload), // ← regulated data, leaving the building
    });
  }
}

Three lines do the damage. bodies ships raw API JSON. stdout carries whatever your app logged, which — per that logs study — is exactly where PII hides. And screenshot sends an image your text scrubber will wave straight through. The model doesn’t need any of it to tell you a locator drifted.

How do you build a redaction gate?

Send the shape of the failure, not its contents, and make the safe path the only path. The model’s job in triage is to cluster and hypothesize — “this is a data-testid rename, that’s a 500 from staging.” None of that needs a payload body. It needs metadata.

Allowlist beats denylist, every time

A denylist asks “what should I strip?” and loses the moment a developer adds a new field you didn’t think of. An allowlist asks “what may I send?” and a new field is invisible to triage until someone deliberately adds it to the safe set. In a regulated codebase, that default — fail closed, not open — is the whole ballgame.

Here’s the core of the gate. Note that the network summarizer never touches a body:

const stripQuery = (url: string) => url.split('?')[0];

// Send the shape of the request, never its payload.
function summarizeRequest(entry: NetworkEntry) {
  return {
    method: entry.method,
    url: stripQuery(entry.url),          // /api/customer?ssn=… → /api/customer
    status: entry.status,
    bodyBytes: entry.responseBody?.length ?? 0,  // size, not content
  };
}

And the text scrubber that runs on every string field before it leaves:

function scrub(text: string): string {
  return text
    .replace(/\b[\w.+-]+@[\w-]+\.[\w.-]+\b/g, '[email]')
    .replace(/\b\d{3}[-\s]?\d{3}[-\s]?\d{3}\b/g, '[sin]')      // Canadian SIN
    .replace(/\b(?:\d[ -]*?){13,16}\b/g, '[card]')
    .replace(/eyJ[\w-]+\.[\w-]+\.[\w-]+/g, '[jwt]');
}

The redacted payload that actually goes out is small, boring, and auditable — which is the point:

const payload = {
  error: scrub(result.error?.message ?? ''),
  console: result.stdout.slice(-5).map(l => scrub(String(l))),
  network: networkEntries.map(summarizeRequest),   // metadata only
  artifacts: { trace: traceUrl, screenshot: shotUrl }, // links, not bytes
};

How do you prove the gate actually holds?

Seed a canary, then assert it can’t escape

You write a test that tries to leak and asserts it can’t. This is the unfakeable senior move — not a policy doc, not a code-review checkbox, but an executable guarantee that runs on every PR. Seed a canary token into a synthetic failure, build the outbound payload through your real redaction path, and assert the canary never appears in the serialized result.

const CANARY = 'CANARY-7Q2F-NEVER-LEAVES-CI';

test('redaction gate never forwards raw data to the LLM', () => {
  // A fake failure whose network body and console contain the canary
  const result = fakeFailureContaining(CANARY);

  const payload = buildTriagePayload(result); // the real gate, not a mock

  const wire = JSON.stringify(payload);
  expect(wire).not.toContain(CANARY);           // body never serialized
  expect(wire).not.toMatch(/@[\w-]+\.[\w]{2,}/); // no raw email survived
  expect(wire).not.toMatch(/eyJ[\w-]+\./);       // no JWT survived
});

When someone later adds a “just include the body, it helps the model” change — and they will — this test goes red before the data ever leaves a runner. I treat this the same way I treat a controlled test environment as a blueprint, not an afterthought: the guardrail is part of the system design, written once and enforced forever, not bolted on after the incident review.

For regulated workloads, the gate is also where you make the model-hosting decision: route triage to a VPC-scoped or self-hosted model, or to a provider endpoint with a signed DPA and ZDR confirmed in writing. The redaction gate shrinks the blast radius; the hosting choice decides who’s standing in it.

The one thing to do today

Open your AI triage code and find the line where data leaves the runner. If it serializes a response body, a screenshot, or raw stdout, you have a redaction gap — wrap that payload in an allowlist summarizer and a scrub() pass, then add the canary test so it can never reopen. Fifteen minutes of work moves you from “shadow AI exfiltration” to “auditable, metadata-only triage,” and it’s the difference between a model that helps you debug and a model that quietly becomes a line item in next year’s breach report.

Want more enterprise-grade patterns for putting AI in a test pipeline without the foot-guns? Get the weekly QA automation insights — no fluff, just battle-tested strategies from 10+ years in the trenches.