Generate Playwright Tests With AI Once, Then Freeze

An AI agent that writes a Playwright spec is doing useful work. The same agent re-deciding how to drive your app on every CI run is a liability you’re paying for in tokens, flakiness, and an audit trail nobody can read. The agent’s job ends at generation. The moment it stays live in your pipeline, you’ve traded a deterministic test suite — the one property that made automated testing worth anything — for a slot machine that bills you per pull.

That’s the whole argument: let the agent author the test, then freeze its output to plain Playwright and run that. Generate once, commit the spec, and your CI runs zero-token deterministic code forever after. Keep the model in the loop at runtime and you’ve rebuilt the flaky, unaccountable suite you were hired to delete.

Why this matters now, not next year

Playwright shipped its Planner, Generator, and Healer agents in v1.56 in October 2025, and adoption moved fast — 72% of QA professionals now use AI to generate tests or optimize scripts, with Gartner projecting 80% of enterprises will use AI-augmented testing by 2027, up from 15% in early 2023. The tooling is here and the org chart already assumes you’re using it.

What nobody put in the rollout deck is where the agent is allowed to run. There’s a seductive demo where the agent explores the app live, figures out the flow, and drives the browser — and teams copy that straight into CI because it looks like the future. It’s the single most expensive mistake you can bake into a pipeline, and it doesn’t show up until you’re three months and a five-figure inference bill deep. The fix is an architectural line drawn in the right place: generation time versus run time.

Why shouldn’t an AI agent run inside your CI loop?

Because an LLM is non-deterministic by construction, and CI’s entire value proposition is determinism. The same prompt against the same model can return different output across runs — not because of temperature, but because of how inference servers batch requests. In September 2025, Thinking Machines Lab traced the primary source of LLM non-determinism to batch variance: reduction kernels produce subtly different numerical results depending on batch size, and since server load fluctuates, batch size shifts unpredictably between identical calls. A follow-up arXiv study measured the same numerical drift in the forward pass even with greedy decoding.

The practical upshot for testers is brutal. When researchers benchmarked models at temperature 0, Claude 4.5 hit 100% determinism on short prompts but lost reproducibility entirely on longer, open-ended ones — and “drive this multi-step checkout flow and assert the total” is about as long and open-ended as prompts get. A live agent in CI means your suite’s behavior is a function of datacenter load. You cannot bisect that. You cannot blame a commit. A test that “fails” might just be the agent rolling a different plan than it did an hour ago.

Then there’s the bill. Re-running an agent on every push, across every worker, on every retry, means you pay inference cost for work you already did once. Teams that get this right report cutting agentic testing costs 40–60% through deterministic caching, model tiering, and trace summarization — and the biggest single lever in that list is deterministic caching, which is just a fancy name for “stop calling the model in CI.” This is the same failure mode I warned about in letting an LLM decide your failures instead of triage them: the model is a phenomenal author and a terrible judge to put on the critical path.

What does “freeze the agent” actually mean?

It means drawing a hard boundary between the agent that writes a test and the runtime that executes it, and never letting the first one touch your pipeline. Freezing is the record half of record-and-replay applied to test authoring: the agent’s reasoning happens once, on a developer’s machine or a dedicated authoring job, and its output — a committed, reviewed .spec.ts file — is the only artifact that ever reaches CI.

Playwright’s own design actually nudges you toward this. The Generator agent’s job is to produce a real spec file with resilient locators, not to drive the browser at runtime. As the official flow describes it, the planner explores the app and writes a Markdown plan, the generator turns that plan into committed spec files, and what runs in CI afterward is ordinary npx playwright test. The agent is scaffolding. You pour the concrete, then you take the scaffolding down.

Think compiler, not interpreter

The mental model that helps: treat the agent like a compiler, not an interpreter. A compiler runs once at build time and emits a deterministic artifact you ship. An interpreter runs your source live, every time, on the user’s machine. You would never ship an interpreter that re-makes decisions in production — and a CI run is your test code’s production.

How do you freeze Playwright Agents output in practice?

You run the agents at authoring time, commit what they emit, review it like any other code, and configure CI to execute only the committed specs. Here’s the concrete sequence.

npx playwright init-agents --loop=vscode
# creates agents/planner, agents/generator, agents/healer + AGENTS.md

test('checkout shows the correct order total', async ({ page }) => {
  await page.goto('/cart');
  await page.getByRole('button', { name: 'Checkout' }).click();

  // AI-generated. STOP HERE: does this assert the real value,
  // or just that *some* total rendered? Pin the number.
  await expect(page.getByTestId('order-total')).toHaveText('$148.00');
});

- name: Run frozen Playwright suite
  run: npx playwright test            # zero-token, deterministic
- name: Guard against live inference in CI
  run: ! grep -rE "anthropic|openai" tests/ --include=*.spec.ts \
       || (echo "LLM call in test path — freeze it first" && exit 1)

The payoff is that your pipeline goes back to being a function of your code, the way it was before AI entered the picture — except you got the authoring speed for free.

What actually changes between a live agent and a frozen one?

The difference isn’t subtle, and it’s worth seeing side by side. The live-agent pipeline optimizes for “the build figures it out”; the frozen pipeline optimizes for “the build tells the truth, repeatably.”

The hidden win is auditability. When a frozen spec fails six weeks from now, a new engineer reads the spec, reads the diff, and knows exactly what was asserted. When a live agent’s run fails, the honest answer to “what did the test actually check?” is “whatever the model decided that morning.” For regulated environments — finance, healthcare, anything that gets audited — that second answer is disqualifying on its own.

Where does the agent legitimately come back in?

At authoring and repair time, behind a human, never auto-firing in the CI loop. The Healer agent is the sharp edge here: it’s built to identify broken locators after a failure and apply fixes automatically, and Forrester credits AI self-healing with cutting test maintenance costs by 50% or more. That’s real value — but only if you run the Healer as an authoring tool that proposes a committed diff a human approves, not as a runtime hook that silently rewrites locators until the build goes green.

Why a self-healer can heal right past a regression

The distinction matters because a self-healer optimizing for “make it pass” will happily heal right past a real regression. I unpack that failure mode in how self-healing locators actually work: a locator that “drifted” and a locator that broke because the feature broke look identical to the Healer. So the rule is the same as the freeze rule — the model can propose a change to your committed source, but a human and a deterministic review gate decide whether it lands. Re-freeze after every heal.

Measure the frozen suite deterministically

And when you do want to validate that the agent wrote good tests rather than just green ones, you measure them deterministically too — run mutation testing against the AI-generated suite to confirm the assertions actually catch injected bugs. That’s a one-time authoring-side check, not a runtime one. The pattern holds everywhere: AI on the authoring side, determinism on the execution side.

Your one concrete next step

This week, audit your pipeline for a single thing: does any CI job make a network call to an LLM during playwright test? Grep your workflow files and your test runtime for an AI SDK import. If you find one, that’s your token leak and your flakiness source in the same place. Move that call to an authoring script a developer runs on purpose, commit the spec it produces, and let CI run frozen code. You’ll cut cost and recover determinism in the same afternoon — and you’ll never have to explain to an auditor why your test suite behaves differently depending on how busy a datacenter was.