Agentic Testing vs Playwright Tests: Who Gets the PR Gate?

My Playwright spec invites a teammate in 193 milliseconds and takes the same three clicks every time. A Claude agent given the same goal needs about 17 seconds and 4 cents per attempt, and it found three different routes to the same result. In the agentic testing vs Playwright tests argument, both are right. They answer different questions, and only one of those questions belongs in the PR gate.

Slack Engineering named the split in June: “Tests enforce journeys. Agents verify goals.” (Slack Engineering). The tempting next step is to treat agent runs as a replacement for brittle scripts. Before you let that idea near a merge button, decide what a red build is supposed to mean.

What did Slack’s agentic testing study actually measure?

Slack ran more than 200 end-to-end workflows on two flows of roughly 15 to 30 steps. It compared three lanes: Claude Sonnet 4.5 driving Playwright MCP, the same model driving the Playwright CLI, and Claude Opus 4.6 generating Playwright tests. Each configuration ran 20 times.

Agent runs cost $15–30 per execution, and only about 20% of runs followed the same normalized action sequence. Slack’s conclusion: deterministic tests stay the CI foundation, and agents add a layer on top for exploration, debugging flaky workflows and reproducing production bugs. I agree with the placement. I disagree with how the 48% is being quoted.

Does Slack’s 48% failure rate mean scripted tests are brittle?

No. It measures a prompt-to-code pipeline, not a reviewed suite. Slack’s generated tests “were generated from loosely specified natural language flows and reused existing page object abstractions,” the agent refined each one until it passed, and no human review is reported. The failures usually came “70-80% through the flow,” on the final interaction or assertion.

That’s exactly the step a reviewer fixes once, before commit. The 48% is an argument for generating the spec with AI once, reviewing it, then freezing it, not for re-deciding the path on every CI run.

What happens when you run the same goal both ways?

On a build with one broken button, the spec failed every run, two lines after the regression. The goal-level agent passed every run, because the goal was still reachable by another route. “Can the owner invite a Viewer?” was still true. “Does the Members page invite still work?” was not.

The demo app has three working routes to the same result: the Invite member button on the Members page, an Invite your team link on the dashboard, and a Quick actions menu. The app logs every click and route change, so each run’s path comes from the app’s evidence, not the agent’s summary. The broken variant has one regression: the Members page button throws TypeError: Cannot read properties of null (reading 'showModal').

The journey, as a spec:

import { test, expect } from '@playwright/test';

test('owner invites a viewer from the Members page', async ({ page, request }) => {
  await request.get('/__reset?run=script');
  await page.goto('/#/members');
  await page.getByRole('button', { name: 'Invite member' }).click();
  const dialog = page.getByRole('dialog', { name: 'Invite a teammate' });
  await dialog.getByLabel('Email').fill('dana@acme.test');
  await dialog.getByLabel('Role').selectOption('Viewer');
  await dialog.getByRole('button', { name: 'Send invite' }).click();

  await expect(page.getByRole('status')).toHaveText('Invitation sent to dana@acme.test');
  await expect(page.getByRole('row', { name: 'dana@acme.test Viewer' })).toBeVisible();
});

The goal, handed to headless Claude Code. mcp.json launches the Playwright MCP server bundled with Playwright 1.65.0 (npx playwright mcp --headless --isolated), and sonnet resolved to Claude Sonnet 5.5.

claude -p "You are testing the web app at http://localhost:$PORT with the Playwright browser tools.
Goal: as the workspace owner, invite dana@acme.test to the workspace with the Viewer role,
then confirm she appears in the members list with the Viewer role.
Only visit http://localhost:$PORT. Note anything that looked broken along the way.
End your reply with exactly one line: VERDICT: PASS or VERDICT: FAIL - <reason>" \
  --model sonnet \
  --output-format json \
  --setting-sources project \
  --strict-mcp-config --mcp-config mcp.json \
  --allowedTools mcp__playwright \
  --max-turns 40

A third lane rewrites the goal line as a journey (“open the Members page, click ‘Invite member’…”) and adds: “Use only that path. If any step on it does not work, stop and report FAIL.”

Same app, three lanes, 100 runs

All 60 agent runs cost $2.69 together. My flow has about five steps against Slack’s 15 to 30, which is why my runs cost cents and theirs cost tens of dollars: turns and accumulated context drive the bill.

On the working build, 15 goal runs used the Members page dialog, 4 took the dashboard link, and 1 jumped from the dashboard straight to the invite form. Nobody opened Quick actions. That’s more repeatable than Slack’s 20%, on a far shorter flow. Every extra decision point adds variance.

The bug report depends on the route

Most broken-build goal runs hit the dead button, noted the TypeError and took another route, the plain-UI cousin of the WebMCP bypass I wrote about. The new problem is the run that took a different route from the start. It went from the dashboard straight to the invite form, never touched the broken button, and reported “I didn’t see anything else broken.” A goal lane can only report bugs on the path it happened to choose.

Agentic testing vs Playwright tests: where does each belong?

Agents sit above the deterministic E2E layer and outside the merge gate. A frozen spec answers “does the journey we promised still work?” in milliseconds. An agent answers “can a user still get there?” and “what looks wrong along the way?” Both agent questions are worth asking, but neither should block a merge. If you constrain an agent until it fails where the spec fails, you’ve rebuilt the spec at about 85 times the runtime.

The last row is real in my data. All 40 goal runs mentioned that the invite form defaults the role to Editor, and 37 flagged that as a least-privilege risk unprompted. 34 noticed that invitees appear as full members with no pending state. I didn’t plant either. My spec selects Viewer explicitly, so it would never notice what the default is, and a default role that grants edit rights is the kind of permission bug that reaches production unseen.

Diff the route against the journey

If you run a goal lane, compare the route each run took against the journey your spec enforces, and flag the runs that left it. Those runs passed through a path the PR gate never exercised.

// usage: node route-check.mjs <path-log.jsonl> "nav-members > members-invite > dialog-submit"
import { readFileSync } from 'node:fs';
const [logFile, journey] = process.argv.slice(2);
const runs = new Map();
for (const line of readFileSync(logFile, 'utf8').trim().split('\n')) {
  const { runId, type, detail } = JSON.parse(line);
  if (type !== 'click') continue;
  runs.set(runId, [...(runs.get(runId) ?? []), detail]);
}
let diverged = 0;
for (const [runId, clicks] of runs) {
  const path = clicks.join(' > ');
  if (path === journey) continue;
  diverged++;
  console.log(`${runId}: took "${path}"`); // the PR gate never ran this path
}
console.log(`${diverged}/${runs.size} runs left the journey`);
process.exitCode = diverged ? 1 : 0;

On the working build it flags 5 of 20 runs. On the broken build it flags all 20: every goal run left the journey. That’s the regression signal the verdict line hid. The WebMCP post gates on which tools the agent called. This gates on which route the app actually saw, the same trajectory idea used to test browser agents, pointed at your own suite.

When does an agent belong in CI at all?

Nightly or before a release, with its output treated as triage input. Slack’s $15–30 per run on real flows is one reason, and the math on AI E2E bills once they run on every pipeline still holds. The other reason is that a goal lane’s green makes a weaker claim than a spec’s green. The choice of driver is its own question, covered in MCP vs CLI vs Playwright’s agents.

Some agent-testing tools pitch replay caching as the thing that makes agent runs cheap enough to gate on. A cached replay is a recorded journey, though: a deterministic test an agent wrote. That’s the frozen-spec model under another name, and it’s fine. Just don’t call it goal verification. A suite that checks a weaker claim than you think is a green suite hiding real bugs.

Start with your top five journeys

Pick the five journeys that would cause an incident if they broke, and make sure each has a reviewed, frozen Playwright spec in the PR gate. Then write one goal prompt per journey for a nightly agent lane, log the routes your app actually sees, and read two things each morning: the runs that left the journey, and the notes. The rest of the QA strategy series builds on that split.

FAQ