Put a Token Budget on Your Playwright AI Agents

The healer agent that fixes a broken locator in three seconds inside your editor will quietly try to fix it forty times at 2 a.m. in CI — and bill you for every attempt. When you ran Playwright’s agents by hand, you were the rate limiter: you watched the loop, you got bored, you stopped it. Wire those same agents into a pipeline and that governor is gone. The first time you’ll notice is the invoice. The fix isn’t a billing alert — it’s a per-PR token budget that fails the build the moment an agent spends past its ceiling.

This is the part of “AI in testing” nobody demos. The conference talks show the green checkmark. They don’t show the ReAct loop reading a full page snapshot, the network log, and the console on every retry, against forty flaky tests, on every push to a busy PR. AI agents burn roughly 50x more tokens than a single-turn chatbot on equivalent work — and a test agent in CI is the most agentic thing you own.

Why this matters now

Playwright shipped its planner, generator, and healer agents as first-class tooling in v1.56, and within a quarter teams stopped running them in VS Code and started chaining them in CI. That’s the right instinct and the expensive one. The pacing mechanism that made them cheap — a human watching — only existed in the editor.

The macro numbers are already ugly. By 2026, 98% of FinOps teams report managing some form of AI spend, and the horror stories are concrete: an audit of 30 engineering teams found one 20-developer org burning $110,000 a month on agent costs, and a single misconfigured 4-agent loop that ran for 11 days and cost $47,000. None of those were test suites — yet. But a healer that “keeps trying to fix the test until it passes” is exactly the unbounded loop those teams got burned by, pointed at your flakiest specs.

Why do Playwright’s AI agents cost so much more in CI than in your editor?

Because the human pacing disappears and the loop grows context on every turn. In your editor you fix one test, watch the agent, and move on. In CI the healer runs each failing test in debug mode — reading console logs, network requests, and page snapshots — then retries until the test passes or it gives up and marks it skipped. Every retry re-sends a growing context window. That’s not linear; agentic ReAct loops accumulate context quadratically as the transcript feeds itself.

Put real 2026 numbers on it. Claude Sonnet 4.6 runs $3 per million input tokens and $15 per million output. A single healer attempt that ingests a 40 KB DOM snapshot, a network log, and the spec is easily 25–40K input tokens before it writes a line. One attempt is cheap. The trap is the multiplication: attempts × failing tests × pushes × open PRs.

The cost isn’t proportional to the value the agent delivers. It’s proportional to how flaky and how busy your suite is. That inversion — paying most where your tests are worst — is why you cap spend at the boundary that maps to value: the pull request.

What should an AI test-agent token budget actually cap?

A single hard ceiling per pull request, denominated in dollars, that aborts the agent run when crossed. Not per-org (too coarse to catch a runaway), not per-test (too granular to administer). The PR is the unit of work a human will eventually review, so it’s the unit you’re willing to spend a fixed amount on.

Token budget enforcement is the set of runtime controls that terminate or pause an agent when it reaches a defined token or cost threshold — distinct from alerts, which only notify after the fact. Three caps earn their keep:

• Per-PR dollar ceiling. The hard stop. When healing a PR’s failures crosses, say, $0.50, the job fails with a clear message and a human looks at why that PR is so broken.
• Per-agent model floor. Triage with a cheap model, escalate only on a real fix. Routing first-pass classification to Haiku instead of Sonnet is a 5–10x unit-cost cut before you optimize anything else.
• Per-page context cap. Bound how much of each DOM snapshot the agent may read. A 40 KB admin page doesn’t need to enter the prompt whole.

How do you enforce a token budget in a Playwright CI pipeline?

Meter every model call, attribute the spend to the PR, and throw the instant you cross the ceiling. Playwright ships no --budget flag, so you wrap the agent the way enterprise SDETs wrap everything that bills by the unit — at the call site.

Build the cost meter

Start with the meter. It’s just the pricing arithmetic and a running total:

// Sonnet 4.6 rates, May 2026: $3 / 1M input, $15 / 1M output
const RATE = { input: 3, output: 15 };
const BUDGET_USD = Number(process.env.PR_TOKEN_BUDGET ?? "0.50");

let spent = 0;

const costOf = (input: number, output: number) =>
  (input * RATE.input + output * RATE.output) / 1_000_000;

export function charge(usage: { input: number; output: number }) {
  spent += costOf(usage.input, usage.output);
  if (spent > BUDGET_USD) {
    // Hard stop: fail the build, don't email a warning after the spend
    throw new Error(`AI budget blown: $${spent.toFixed(2)} > $${BUDGET_USD}`);
  }
}

Wrap the healer loop in the meter

Then wrap the healer loop so the cheap model triages, the context is bounded, and every call charges the meter:

for (const failing of redTests) {
  const result = await runHealer(failing, {
    model: "claude-haiku-4-5",   // triage cheap; escalate only on a real fix
    maxSnapshotBytes: 50_000,    // cap the DOM the agent reads per page
  });
  charge(result.usage);          // throws the moment the PR budget is gone
}

The four-step enforcement pipeline

The full pipeline is four steps, and the discipline is in the metering, not the cleverness.

Where do you cut tokens without losing healing power?

You cut the inputs the agent doesn’t need and route work to the cheapest model that can do it — none of which weakens the fix. Four levers, in order of payoff:

First, prompt caching. Stable context — your test conventions, page-object docs, the framework prompt — reads from cache at 10% of the standard input rate on repeat calls. A healer run hammering the same suite is the ideal cache hit pattern. Second, model routing: Haiku triages whether a failure is a locator break or a real regression; Sonnet only gets the ones worth a real fix. Don’t make an LLM your gate when a cheaper one can triage the failure first. Third, context capping — the maxSnapshotBytes ceiling above, plus stripping noise from traces before they ever reach the model (which you should be doing anyway to avoid leaking PII to the LLM). Fourth, batch the non-urgent work: nightly eval runs and bulk regeneration go through the Batch API at a 50% discount because nothing is waiting on them.

There’s a second-order win here. A visible per-PR cost turns flakiness into a line item. The team whose suite costs $4 to heal on every push stops blaming the agent and starts fixing the tests that hide real bugs behind green retries. And before you trust the healer’s fixes at all — budgeted or not — measure its precision in shadow mode, because a cheap wrong fix is still a wrong fix.

Your one next step

This week, add the charge() meter to a single CI job — pick your flakiest suite — and run it in report-only mode. Don’t fail any builds yet. Just log the dollar cost per PR for two weeks. You’ll learn two things you don’t currently know: what your AI testing actually costs per merge, and which three specs are responsible for most of it. Set the ceiling at p95 after that, and flip enforcement on. The budget pays for itself the first night a flaky deploy would have looped the healer into the four figures.