Your Playwright Generator Is Only as Good as Its Seed

The Playwright generator agent will happily write you a passing test that imports nothing from your framework, inlines a CSS selector that breaks on the next design tweak, and asserts toBeVisible() on a total that’s supposed to read $129.00. Then your team blames the model and spends a week tuning prompts. The model isn’t the problem. The context you handed it is — and unlike the prompt, the context is something you build once and commit.

I’ve stood up test-generation pipelines on enterprise suites where a single bad convention, multiplied across 300 generated specs, becomes a quarter of cleanup. The teams that get clean, maintainable output from the generator aren’t writing better prompts each run. They’ve engineered three files the agent reads on every invocation. That’s the whole game, and almost nobody talks about it.

Why prompt-tuning the generator is the wrong layer

Playwright shipped its three Test Agents — planner, generator, and healer — into the framework starting in v1.56. The generator’s job is narrow: read the planner’s Markdown plan, open the running app, verify locators live against the real DOM, and emit spec files. Done well, it produces role-based locators that are genuinely stable. Hand-written role locators flake at well under 2% in disciplined suites, and the generator lands in the same range — when it follows your patterns.

The failure isn’t random. Weaker or cheaper models — the ones you actually run at scale because you can’t afford a frontier model on every CI generation — drift from your conventions in predictable ways. They inline selectors instead of importing page objects, skip your fixtures, drop files in the wrong directory, and write optimistic assertions that pass on a happy-path render but say nothing about correctness.

This is the same lesson I argued in MCP vs CLI vs Agents in 2026: the integration surface is an architectural decision, not a per-run choice. Generation is no different. You fix it at the layer that persists.

What context does the generator actually read?

When you invoke the generator, Playwright assembles a context payload from three sources you control, plus the live DOM. Knowing exactly what goes in lets you stop guessing.

The agent definition is a Markdown file in agents/ (for the generator, playwright-test-generator.agent.md) that ships when you run init-agents. It’s the system-level instruction set — your house rules. The seed file (seed.spec.ts) is copied into the context and into generated tests; it’s where your fixtures, auth, and base setup live. The plan is the planner’s output, which the generator treats as the spec to implement.

Most teams run init-agents, accept the blank seed, never open the agent definition, and then wonder why output quality is a coin flip. The defaults are a starting point, not a standard.

The blank seed: where inline auth and CSS selectors come from

Leave seed.spec.ts blank and the generator has no idea how a logged-in session exists in your app. So it invents one — inline, in every spec, using whatever it can see in the DOM.

test('order total after checkout', async ({ page }) => {
  await page.goto('https://staging.example.com/login');
  await page.fill('#email', 'qa-user@example.com');
  await page.fill('input[name="password"]', 'hunter2');
  await page.click('.btn-primary.login-submit'); // brittle CSS
  await page.goto('https://staging.example.com/checkout');
  await page.getByRole('button', { name: 'Place order' }).click();
  await expect(page.locator('.order-total')).toBeVisible(); // says nothing
});

import { test, expect } from '../fixtures/authed';

test('order total after checkout', async ({ authedPage }) => {
  await authedPage.goto('/checkout');
  await authedPage.getByRole('button', { name: 'Place order' }).click();
  await expect(authedPage.getByTestId('order-total')).toHaveText('$129.00');
});

The left version isn’t wrong because the model is dumb. It’s wrong because the only context it had was the DOM, so it reverse-engineered a login flow from form fields and hardcoded a staging URL and a password into a spec file. The right version exists because the seed told it authedPage is how you get a session.

The seed file that fixes it

Here’s the seed that produces the right column:

import { test as base } from '@playwright/test';

// The generator copies this into context AND into generated specs.
// Anything you want every test to assume must live here or in a fixture
// this file imports. A blank seed forces the model to improvise.
export { test, expect } from './fixtures/authed';

The agent definition: your locator standard, in writing

The seed handles setup. The agent definition handles style — and this is where you encode the locator discipline I broke down in locator vs getByRole. The generator already prefers role-based locators, but “prefers” isn’t “is required to.” Make it a rule.

# Test Generator — House Rules

You generate Playwright specs for this repository. Follow these without exception:

- Locators: getByRole or getByTestId ONLY. Never CSS, never XPath, never raw text.
- Page interactions: import page objects from `tests/pages/`. Never inline selectors.
- Assertions: assert on values, not visibility. Prefer toHaveText / toHaveValue
  over toBeVisible for any element that displays data.
- Setup: assume the `authed` fixture from the seed. Never write a login flow inline.
- File placement: specs go in `tests/specs/<feature>/`, mirroring the plan.

That assertion rule is the one that earns its keep. A generator left to its own judgment writes optimistic assertions that pass on render and verify nothing — the exact mechanism behind green tests that hide real bugs. toBeVisible() on an order total is a green checkmark over an untested calculation. Forcing toHaveText means the generated test actually has an opinion about correctness.

Putting it together: a generation setup that scales

This is the sequence I run before letting the generator touch a real repo. It’s a one-time investment that every future generation inherits.

Enforce the conventions with a CI check

That last gate is cheap and worth it:

# Fail generation PRs that smuggle in banned patterns
if git diff origin/main..HEAD -- 'tests/specs/**' \
  | grep -qE '^\+.*page\.locator\(|^\+.*\.click\(.[.#]'; then
  echo "::error::Generated spec used a raw CSS selector — fix the agent definition"; exit 1
fi

Fix the context, not the spec

Notice the error message points at the agent definition, not the spec. When generated output is wrong, you don’t hand-fix the spec — that’s a one-time patch that the next generation re-breaks. You fix the context file so every future spec inherits the correction. That feedback loop is the entire discipline.

What I’d do differently

Early on, I treated each generation as a conversation — generate, eyeball the output, re-prompt “use page objects this time,” generate again. It felt productive and it scaled to exactly zero. The output quality tracked my patience, not my standards. The moment I moved every correction out of the prompt and into the seed and agent definition, generation went from a babysitting task to a build step. If you find yourself typing the same correction into the generator twice, stop — that correction belongs in a committed file. The generator is downstream of self-healing too; understanding how self-healing locators actually work tells you why a well-seeded role locator survives a UI change that kills an inline CSS one.

The one thing to do today

Open your seed.spec.ts. If it’s blank — and after a default init-agents it is — that’s your highest-ROI fix. Wire it to your real auth fixture and base config, then add five locator-and-assertion rules to playwright-test-generator.agent.md. Regenerate one spec you’ve already seen the messy version of and watch the inline auth and CSS selectors disappear. That’s 20 minutes of context engineering that every future generation inherits, versus an indefinite tax of re-prompting the same fixes forever. Browse the rest of the AI-powered testing pillar for where agents earn their place — and where they quietly cost you.

If you want field-tested takes on running AI agents inside real enterprise suites — not the demo version — that’s what I publish here every week.