Our AI E2E Suite Cost $400 a Day in CI

Our AI-driven E2E agent spent 9.6 seconds thinking and 1.1 seconds clicking. Per step. Eleven steps to book a flight, eighteen flows in the pilot, thirty-two pipeline runs a day. On a laptop that ratio reads as magic — the thing figured out the date picker by itself. Billed across a parallel grid at a mid-size online travel agency, the magic turned into a line item: our LLM E2E test CI cost hit $403 a day, and I had to stand up and explain it.

Here’s the sentence that ended the pilot, and it’s the whole post: we were not paying to test the booking flow. We were paying to re-discover it, from scratch, thirty-two times a day.

Why LLM E2E test CI cost breaks the moment you leave the laptop

Agent-driven E2E is genuinely good at one job — walking into an app it has never seen and figuring out how to drive it. That’s exploration, and it’s worth paying for exactly once per flow. The demo economics are seductive because the demo runs the agent once. Your CI pipeline runs it on every push, on every worker, on every retry, forever.

The vendors know this, and a couple of them say so out loud. Microsoft’s own Playwright MCP README concedes that coding agents increasingly prefer CLI workflows because MCP invocations load “verbose accessibility trees” into the model’s context, and that MCP earns its keep only “where maintaining continuous browser context outweighs token cost concerns.” In a PR gate, it never outweighs it. Stagehand’s founder went further on Hacker News: “we don’t really intend for Stagehand to be used in a test suite… If you’re doing QA on your own site, base Playwright is likely the better solution.” When the people selling the agent tell you not to put it in your suite, believe them — and note that which integration layer you pick decides your token bill before you write a single test.

Where does the money actually go in an LLM E2E test?

Almost none of it goes to the browser. It goes to re-sending the page to the model. The Anthropic Messages API is stateless — you send the full conversation on every call — so on step 11, the agent re-transmits the accessibility snapshots from steps 1 through 10 just to decide where to click next. Cost grows with the square of the flow length, not with the number of assertions.

The reasoning tax you don’t see in a demo

Every step pays a fixed toll before the browser moves: serialize the page, ship it, wait for the model to reason, parse the action. We measured a mean of 9.6 seconds of model latency against 1.1 seconds of actual browser work. That’s an 8.7:1 tax, and it’s invisible at demo scale because one flow taking two minutes instead of twenty seconds just looks like a slow app.

Then we measured the snapshot itself. There is no published number for how many tokens a Playwright accessibility snapshot costs, so we counted ours:

import Anthropic from '@anthropic-ai/sdk';
import { chromium } from '@playwright/test';

const client = new Anthropic();
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://staging.example.com/search?from=YYZ&to=LHR');

// _snapshotForAI() is what Playwright MCP feeds the model
const snapshot = await (page as any)._snapshotForAI();
const { input_tokens } = await client.messages.countTokens({
  model: 'claude-sonnet-5',
  messages: [{ role: 'user', content: snapshot }],
});
console.log(`snapshot = ${input_tokens} tokens`);

6,100 tokens for one search-results page. The single worst offender was the departure-date calendar: 2,300 of those tokens were the date grid, because every selectable day is its own button node in the accessibility tree. We were paying the model to read a month of Tuesdays every step of every flow in every pull request.

Because every step re-sends every prior snapshot, the context you pay for is triangular, not linear. Here’s the same 6,100-token snapshot compounding as a flow gets longer:

Doubling a flow’s length doesn’t double its bill — it roughly quadruples it. Hold onto that row for 17 steps; it’s about to explain why our first cost fix backfired.

Eighteen flows at $0.70 is $12.60 a suite run. Thirty-two runs a day is $403. For scale, the 8-core runners executing all of it billed $0.022/minute — about $29 a day. The compute was a rounding error. The thinking was the bill.

Why don’t a cheaper model or prompt caching fix LLM E2E test CI cost?

Because both attack the price per token, and the problem is the number of tokens — which is set by architecture, not by procurement. We tried three cost fixes before we touched the design, and all three failed in ways worth stealing.

Dead end 1: drop to a cheaper model

Haiku 4.5 runs $1/$5 per MTok against Sonnet’s $3/$15, so on paper this was a 67% cut. In practice the cheaper model needed more steps to reach the same goal — our median flow went from 11 steps to 17. Look back at that table: 17 steps is 25x the raw context of a 3-step flow, against 11x for eleven. The extra six steps ate the discount before it reached us. Haiku also misread the date picker and started booking the wrong month, pushing our false-failure rate from 3% to 22%, and we spent more engineer-hours triaging phantom failures than we saved in tokens.

We weren’t imagining it. When Slack’s engineering team benchmarked agentic E2E across 200+ runs, they found Haiku consumed more tokens than Opus on the same complex flow — 5.7M against 3.8M — precisely because it retried more. A cheaper model that takes more turns is not a cheaper model.

Dead end 2: turn on prompt caching

We already had caching on, which is why the “just cache it” advice stung. Caching gives you a 90% discount on cache reads for a 1.25x write premium, and it does cache the stable prefix — tool schemas, system prompt. But it cannot cache the thing that’s killing you, because a fresh page snapshot arrives as new bytes at the end of the conversation on every single step.

Then we found the real trap. Checking usage.cache_read_input_tokens on our step requests, roughly 40% came back zero — no error, no warning, just full price. Prompt caching walks back at most 20 content blocks from a breakpoint to find a prior cache entry, and a tool-heavy agent turn emits enough tool_use/tool_result pairs to blow straight past that window. Fixing our breakpoint placement got hit rates to ~85% within a flow and brought the bill to $310 a day.

Dead end 3: run it less often

Moving the agent suite to a nightly job took it to about $25 a day. It also moved our feedback loop from seven minutes to fourteen hours, which is a retreat dressed as an optimization. If a gate only tells you about the breakage tomorrow, you don’t have a gate. The same logic that keeps performance tests out of the PR pipeline doesn’t rescue you here: unlike a load test, an E2E booking check really does belong on every PR. The test was right. The executor was wrong.

The fix: move the model from run time to author time

Let the agent explore the flow once, emit a real Playwright spec, and let CI run the spec. Playwright already ships this pipeline — planner, generator, and healer agents — and the generator’s entire job is to produce a committed .spec.ts with verified locators, not to drive the browser at 3 a.m.

The change that mattered was organizational, not technical: the agent got its own job, and that job is not the PR gate.

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      # The PR gate runs committed specs. No model, no API key, no tokens.
      - run: npx playwright test --workers=6
        env:
          BASE_URL: ${{ secrets.STAGING_URL }}

      # Fails the build if anyone smuggles a live model call back into tests/
      - name: No LLM calls in the PR gate
        run: |
          ! grep -rInE "@anthropic-ai/sdk|@playwright/mcp" tests/

That last step is the load-bearing one. Freezing an agent’s output is easy; keeping it frozen is a policy problem, because the first time a spec breaks, somebody’s instinct is to wire the healer back in to fix it live. Grep is a cheaper guardrail than a budget alert. If you do want the model in CI, cap it deliberately with a per-PR token budget that fails the build rather than leaving it uncapped.

Freezing is not free, and most posts won’t tell you that

Here’s the part that should give you pause, including in my own recommendation. That same Slack study found the generated Playwright tests failed 48% of the time on complex flows, while the live agent failed only 12%. Compiling away the model also compiles away its ability to recover — the agent routes around a moved button, and the frozen spec just dies on it.

Which makes the review gate the whole job. When a generated spec lands, somebody has to read it and ask the questions the agent never will: did it assert on a real network response, or on a toast that renders whether or not the booking saved? Did it paper over a missing loading state with a bare waitForTimeout(3000)? Did it quietly hit the API directly instead of clicking through the UI it was supposed to be testing? Four of the first eighteen specs the generator handed us went straight back on exactly those grounds. Freeze an unreviewed spec and all you’ve committed is one plausible afternoon, preserved forever.

Authoring now runs on demand — when a flow actually changes — at roughly $14 a week, and a human reviews the diff before it lands.

Results

The number I care about most isn’t the ~99% cost cut. It’s that the cost became predictable. Token spend on the same task varied by up to 10x run to run depending on how many wrong turns the agent took, and a step with 10x cost variance has no business behind a gate that fires on every commit, no matter how good the average looks. Magnitude was never really the problem — we could have absorbed $403 a day if we could forecast it. What we couldn’t do was tell finance what next month looked like.

What I’d do differently

Measure the snapshot before writing the first flow. Twenty minutes with countTokens() would have told us the date picker was 2,300 tokens and that this architecture could never be cheap. We instead found out at $403 a day, in a sprint review.

Instrument cost per run from day one, not per month. Our billing dashboard aggregated monthly, so the pilot looked fine for three weeks and then didn’t. Cost per pipeline run is the only unit that scales the way your CI does — and it’s the one that would have flagged this while it was still a $30/day problem.

I’d still run the pilot. The agent found two real bugs during exploration that our scripted suite had missed for a year, including a date picker that silently rejected February 29. Exploration earned its keep. Execution never did.

Your one next step

Take your single most expensive agent-driven flow and print its billable tokens per run. If it’s over 100K, you don’t have a testing tool in your pipeline — you have a subscription to re-learning your own app. Author the spec once, commit it, and let the model go do something a script can’t.