Test What Your AI Agent Does, Not What It Says

Your product’s new AI assistant just told the user “Done — I’ve booked your meeting for Tuesday at 2pm.” Your Playwright test sees that sentence, expect(reply).toContainText('booked') goes green, and CI is happy. Meanwhile the agent called the calendar API with the wrong timezone, or picked the create_event tool when the user asked to reschedule, or didn’t call any tool at all and just hallucinated the confirmation. Your test passed. Production is broken. The gap between what an agent says and what it does is the single most dangerous blind spot in testing agentic features, and asserting on the reply text walks you straight into it.

The fix is a boundary change. Stop pointing your assertions at the chat bubble and start pointing them at the tool call — the structured { name, arguments } the agent emits when it decides to act. That’s where the real behavior lives, it’s deterministic in a way the prose never will be, and it’s the exact boundary your production observability stack is already watching.

Why is this suddenly every SDET’s problem?

Because agents stopped being demos and started being the feature you’re on the hook to test. In LangChain’s State of Agent Engineering survey of 1,340 practitioners (Nov–Dec 2025), 57% already run agents in production, and Gartner projects that 40% of enterprise applications will embed task-specific AI agents by the end of 2026, up from under 5% in 2025. The most common agent use case in that survey was customer service (26.5%) — agents put directly in front of users, taking real actions on real accounts.

An agent that only talks fails in ways your existing tests catch. An agent that acts — books, cancels, refunds, emails, edits a record — fails in ways a text assertion is structurally blind to. And the “acting” agent is now the norm. The failure mode moved; most test suites didn’t move with it.

I watched this land at a fintech client last year. Their support agent could issue account credits. The E2E suite asserted the confirmation copy — “Your $50 credit is on the way” — and stayed green for six weeks. The bug: on a specific account tier the agent called issue_credit with the amount in dollars where the API expected cents, so a $50 credit posted as $0.50. No text assertion on Earth catches that. A single assertion on the tool-call arguments catches it on the first run.

Why does asserting on the reply hide the worst bugs?

Because “said the right thing” and “did the right thing” are independent variables, and tool-calling is where they diverge. The model can produce a perfectly plausible confirmation while selecting the wrong tool, passing a malformed argument, or skipping the call entirely. This isn’t a rare tail case — it’s the documented failure taxonomy of tool-use models.

The Berkeley Function-Calling Leaderboard, the standard benchmark for this, exists precisely because even frontier models make measurable errors in which function they call and what arguments they pass. Research like ToolScan characterizes the recurring error classes — wrong tool selected, required argument omitted, argument hallucinated, value out of range. These are the bugs your users feel, and every one of them can coexist with a flawless-looking reply.

There’s a telling blind spot in the industry here. In one 2025 survey of engineering leaders, only 5% cited accurate tool calling as a top technical challenge — not because it’s reliable, but because most teams aren’t looking at it. They’re looking at the chat transcript, same as their tests are.

Where is the right assertion boundary?

At the tool-call edge: the moment the agent’s backend turns a model decision into an outbound request. In almost every architecture that edge is observable from the browser as a network call — POST /api/agent/tools/create_event or a batched /api/agent/step payload that names the tool and carries its arguments. That’s the seam Playwright is built to sit on.

This isn’t a novel idea in production — it’s already standard for observability. In the LangChain survey, 89% of teams have implemented agent observability and 62% have detailed tracing to inspect individual tool calls. The insight is simply to assert in test on the exact boundary your ops team already traces in prod. If tool calls are the truth in production, they’re the truth in CI too.

This is a different boundary from the one I argued for in testing an AI feature’s output contract. That post is about asserting the shape of what the model returns to the UI — validate the JSON, not the prose. This one is about asserting the action the agent takes on the way there. You want both: the contract check guards the response the user sees, the tool-call check guards the side effect the user’s account suffers. They fail on different bugs.

How do you assert on a single tool call in Playwright?

Intercept the request at the tool boundary, capture the payload, and assert on name and arguments with the same rigor you’d apply to any API contract test. Playwright’s request interception gives you the call without touching the UI.

Capture the call and pin the response

Start by catching the tool request and stubbing a deterministic response so the rest of the flow is stable:

import { test, expect } from '@playwright/test';

test('agent books with correct args', async ({ page }) => {
  let toolCall: any = null;

  await page.route('**/api/agent/tools/create_event', async (route) => {
    toolCall = route.request().postDataJSON();      // capture the call
    await route.fulfill({ json: { ok: true, id: 'evt_123' } }); // pin the response
  });
  // ...drive the UI below
});

Assert on the arguments, not the copy

Now drive the feature and assert on the arguments, not the confirmation copy. Validate the structured call with a schema so a shape drift fails loudly:

import { z } from 'zod';

const CreateEvent = z.object({
  title: z.string().min(1),
  startsAt: z.string().datetime({ offset: true }), // timezone-aware, non-negotiable
  durationMin: z.number().int().positive(),
});

await page.getByRole('textbox').fill('Book a 30-min sync Tuesday 2pm');
await page.getByRole('button', { name: 'Send' }).click();
await expect.poll(() => toolCall).not.toBeNull();

const args = CreateEvent.parse(toolCall.arguments);   // fails on bad shape
expect(args.durationMin).toBe(30);                    // fails on the cents-vs-dollars class of bug

The .datetime({ offset: true }) check alone would have caught my fintech client’s timezone bug. The durationMin assertion catches the wrong-units class. Neither depends on a single word the model chose to say.

How do you test a multi-step agent flow?

Pin a golden tool-call transcript — the ordered sequence of { name, arguments } the agent should emit for a task — and assert the run against it. A single tool call is easy; the interesting agent bugs live in the choreography: it calls search_flights then book_flight but skips check_price, or it loops get_status eleven times because it never advances state.

const calls: { name: string; arguments: any }[] = [];
await page.route('**/api/agent/tools/**', async (route) => {
  const { tool, arguments: args } = route.request().postDataJSON();
  calls.push({ name: tool, arguments: args });
  await route.fulfill({ json: stubFor(tool) });
});

await runBookingTask(page);

const sequence = calls.map((c) => c.name);
expect(sequence).toEqual(['search_flights', 'check_price', 'book_flight']); // order is the contract
expect(calls.length).toBeLessThanOrEqual(6);                                 // loop ceiling

Doesn’t this just recreate flaky tests at a new boundary?

No — and that’s the whole point of moving the assertion. The reply text is non-deterministic because it’s generated token by token; I’ve written before about how temperature 0 doesn’t save you there. The tool boundary is different: by stubbing the tool responses with route.fulfill, you remove the downstream randomness and make the agent’s decisions the only variable under test. You’re testing decision logic against fixed inputs, not racing a live backend.

Two rules keep it stable. First, assert on the smallest set of arguments that encode correctness — the amount, the recipient, the timezone — and ignore the rest; a golden transcript that pins every field is a golden transcript that breaks on the next harmless prompt tweak. Second, give your goldens an owner and a review cadence. When a genuine product change alters the expected tool sequence, updating the golden should be a deliberate diff a human approves, not a --update-snapshots reflex. Treat it like any other contract: a change to it is a change to behavior.

What this doesn’t replace

Assert on tool calls, and you still need the output-contract check on the user-facing response, plus at least one real end-to-end run against a live model in a nightly lane to catch the decisions your stubs can’t. The tool-call assertion is the deterministic core that belongs on every PR; the live eval is the slower guardrail that runs less often. What you’re retiring is the brittle middle — the toContainText assertion on generated prose that was never testing behavior in the first place.

Your one next step this week

Pick the single highest-stakes action your agent can take — the one that moves money, deletes data, or emails a customer. Add one page.route intercept on its tool boundary and one assertion on its arguments. Not the whole suite, not every flow — one call, the one that would hurt most if it went wrong silently. You’ll likely find, as I have on every agent codebase I’ve touched, that the reply was green and the arguments were not.

If you want the next pattern in this series — building the golden-transcript harness and the nightly live-eval lane around it — get the deep dives as they land.