Stop Asserting Clicks: Trajectory Tests for Browser Agents

Your product just shipped a browser agent — the kind that reads the page, decides what to click, and drives a checkout or a booking on the user’s behalf. You write the obvious end-to-end test: launch the agent, then expect it to click #tab-2, then #confirm-btn, then land on /success. It’s green today. Tomorrow the model reorders two steps, clicks a differently-labelled button that does the same thing, and your test is red — even though the agent did its job perfectly. Assert on the exact click sequence and you’ve built a test that fails every time the agent succeeds differently.

The fix is to stop testing the path and start testing the trajectory: the sequence of decisions, judged against invariants instead of exact steps. Did the agent reach the goal state? Did it avoid the actions it must never take? Did it get there inside a step budget? That’s what you can actually assert on a non-deterministic system without your CI turning into a coin flip.

Why do exact-step assertions fail on agents that work?

Because the agent’s output is non-deterministic at the token level, and every click is downstream of a token. You cannot pin the path even if you want to. This isn’t a temperature knob you forgot to turn to zero — it’s structural.

The Thinking Machines Lab team generated 1,000 completions from the same prompt at temperature 0 and got 80 unique outputs; the completions were identical for the first 102 tokens, then diverged at the 103rd. The broader literature on “deterministic” LLM settings traces this to batch-dependent floating-point reductions, GPU kernel non-associativity, and mixture-of-experts routing — none of which you control from a test file. A browser agent inherits all of it and adds environment noise on top: a banner that loads late, an A/B bucket, a slightly different DOM order.

So the flake isn’t in your infrastructure; it’s in the shape of what you’re asserting. The path is a random variable. The outcome and the safety envelope are the properties you actually care about, and those are far more stable than any single trajectory through them.

What is a trajectory test, concretely?

A trajectory test records the agent’s decision stream — each { step, observation, action, target } tuple — and asserts on properties of that stream rather than on any exact member of it. Think of it as the difference between asserting “the agent typed these 14 keystrokes” and asserting “the agent ended up logged in, never touched the delete button, and did it in under 12 steps.”

This is exactly the move the research benchmarks made when they hit the same wall. WebArena scores its 812 tasks by success rate on the generated trajectory — outcome, not path. And the COLM 2025 “Illusion of Progress” study showed why loose path-matching is dangerous in the other direction: static benchmarks with sloppy scoring overestimated live-web agent competence badly, and their fix, WebJudge, reaches ~85% agreement with human annotators — 8 to 18 points higher than prior methods — precisely by checking that key intermediate steps were retained rather than accumulating screenshots. The lesson cuts both ways: assert too tightly and you get flake; assert too loosely and you pass broken agents. Trajectory invariants are the middle.

I learned this the expensive way on a logistics client’s shipment-booking agent. Our first suite asserted the click path. Over three weeks it went red 11 times; every single one was a false alarm — the agent had found an equivalent route through the UI. We were the flakiest suite in the org and we’d caught zero real bugs. The rewrite asserted four invariants instead. It caught a genuine regression in week one: after a prompt tweak, the agent started calling the address-autocomplete widget twice, blowing the step budget on 6% of runs. A path assertion would never have seen it; a step-budget invariant caught it immediately.

Which invariants actually earn their place?

Four categories cover the vast majority of what you want to guarantee. Each is stable under non-determinism because none of them names a specific step.

Notice what’s missing: any assertion about which selector, in which order, with which exact arguments. That granularity belongs one layer down, at the single tool call — which is a different and complementary technique I covered in testing what your AI agent does, not what it says. Trajectory tests sit above it, guarding the shape of the whole run.

How do you make this deterministic and cheap in CI?

Record once, replay forever. The reason agent E2E tests are slow and expensive is that they call a live model on every run — and a non-deterministic one, so you’re paying for flake. Capture the decision stream on a real run, freeze it as a fixture, and replay the frozen trajectory against your invariant assertions in the PR gate. No model call, no network, no flake, sub-second.

import { test, expect } from '@playwright/test';
import { replayTrajectory } from './harness';
import booking from './fixtures/book-shipment.trajectory.json';

test('booking agent stays inside its envelope', async ({ page }) => {
  // Replays a recorded decision stream — no live LLM call in the PR gate
  const run = await replayTrajectory(page, booking);

  // 1. Goal state
  await expect(page).toHaveURL(/\/shipments\/[A-Z0-9]{8}\/confirmed/);

  // 2. Forbidden actions — the safety envelope
  const forbidden = ['delete_shipment', 'issue_refund', 'submit_admin_form'];
  const usedTools = run.actions.map(a => a.tool);
  expect(forbidden.filter(f => usedTools.includes(f))).toEqual([]);

  // 3. Step budget
  expect(run.actions.length).toBeLessThanOrEqual(12);

  // 4. Key waypoints, order-free
  expect(run.states).toEqual(expect.arrayContaining(['rate_selected', 'address_confirmed']));
});

The recorded fixture is the same artifact your production observability stack already emits. In LangChain’s State of Agent Engineering survey, 89% of teams run agent observability and 62% capture detailed tool-call traces — so the decision stream you need to record is data you’re very likely already collecting in prod. You’re not building new plumbing; you’re pointing a test at an existing trace format.

Recording a fixture you can trust

A replay is only as good as the trajectory you froze, so record against a pinned environment — seeded database, fixed feature flags, a specific model version — and capture two or three runs, not one. If the runs disagree on their path that’s fine and expected; if they disagree on an invariant (one reaches the goal, another doesn’t), you’ve found a reliability bug before you’ve written a single assertion. Strip PII from the observations before you commit the fixture, the same way you would redact Playwright traces before they reach an LLM.

Where trajectory tests stop, and something else takes over

Trajectory invariants tell you the agent stayed in bounds and hit the goal. They deliberately say nothing about quality — whether the summary it wrote was accurate, whether the rate it picked was the cheapest. That’s a judgment call, and it belongs to an eval, not an assertion. When you need to grade the content of what the agent produced, reach for a rubric-scored evaluator, with all the calibration headaches that entails — which is its own discipline I dug into in why LLM-as-judge tests go flaky. Keep the two separate: invariants gate the PR, evals grade the quality.

The mindset shift

The hard part of testing agents isn’t tooling — Playwright already gives you everything you need to observe actions and states. The hard part is unlearning the deterministic-E2E instinct that says a good test pins exact behavior. With an agent, pinning exact behavior is the bug in your test. You’re testing a sampler, and the only honest assertions are the ones that hold across samples: the goal it must reach, the lines it must not cross, the budget it must respect.

Your concrete next step: take your flakiest agent test, look at what it asserts, and count how many of its assertions name a specific selector or step index. Every one of those is a future false alarm. Replace them with the four invariants above, record one clean trajectory as a fixture, and move the “is the answer good?” question into a separate nightly eval. You’ll trade a suite that cries wolf for one that actually guards the envelope.