Your Agent Tests Pass Because Its Tools Never Fail

Here’s the uncomfortable truth about your agent test suite: every test you have runs against tools that work. The calendar API returns 200, the payment service settles, the search index answers in 40ms. So the only behavior you’ve ever verified is the happy path — the one path production is least likely to keep serving you. The interesting question isn’t “does the agent book the meeting when the API works?” It’s “what does the agent tell the user when the API times out?” And almost nobody tests that, which is exactly why agents ship a specific, dangerous bug: they narrate success over a failed tool call.

Stop testing agents against a perfect backend. Inject faults at the tool boundary — timeouts, 5xxs, malformed payloads, schema-valid-but-wrong responses — and assert that the agent recovers honestly instead of hallucinating a confirmation. That’s the test that separates a demo from a product.

Why is the failed-tool path the one that actually breaks?

Because that’s where the agent has the most freedom and the least supervision. When a tool succeeds, the model has a clean result to summarize. When a tool fails, the model has to decide — retry, pick a fallback, or tell the user it couldn’t. Left to its own devices, an LLM’s most fluent option is often to keep talking as if everything worked. That’s not a rare glitch; it’s the structural incentive of a next-token predictor rewarded for plausible-sounding replies.

The benchmarks make the failure rate concrete. On τ-bench, Sierra’s benchmark for agents operating in dynamic tool environments, even frontier models complete fewer than half of the retail and airline tasks reliably — and the consistency metric (pass^k across repeated trials) collapses far faster than the single-run pass rate, meaning agents that succeed once routinely fail the same task on a retry. The Berkeley Function-Calling Leaderboard shows the same models make measurable errors in argument construction even when the tool is available and healthy. Now degrade the tool and you’ve stacked a second failure surface on top.

I hit this head-on at a logistics client last year. Their support agent could reschedule a delivery by calling a reschedule_shipment tool. In staging, everything green. In production, the downstream scheduling service had a p99 of 9 seconds and occasionally returned a 504. When it did, the agent replied “All set — your delivery is now booked for Thursday” and called nothing else. Customers showed up Thursday to no delivery. The tool had 504’d; the agent invented the confirmation. Our entire E2E suite missed it for eight weeks because not one test had ever made that tool fail. The fix took an afternoon. The trust took a quarter to rebuild.

What does “recover honestly” actually mean?

It means the agent’s observable behavior after a tool failure matches one of three acceptable outcomes — and never the fourth. This is the contract you’re testing, and writing it down is half the battle because most teams have never specified it.

The three acceptable outcomes: retry the same tool with backoff (and eventually succeed or give up), fall back to an alternative tool or a degraded path, or surface the failure to the user with an honest, actionable message. The one unacceptable outcome — the bug — is hallucinated success: a confirmation message with no successful side effect behind it. Google’s SRE guidance on graceful degradation frames the same principle for services: a degraded-but-honest response beats a confident-but-wrong one every time. Agents need the identical discipline, and it has to be tested, not hoped for.

This builds directly on a boundary shift I’ve argued before: assert on the tool calls your agent makes, not the text it says. Fault injection is the natural next move — once you’re watching the tool boundary, you can also control it.

How do you inject faults at the tool boundary in Playwright?

You interpose a controllable fault layer between the agent and its tools, then drive it per-test. In most architectures the agent’s tool calls surface as network requests — either directly from the browser or from a backend endpoint you can front with a proxy. Playwright’s page.route handles the browser-observable case cleanly; for server-side tools, a header-driven fault mode on a test build does the same job.

Step 1: Catalogue the faults worth injecting

Start by cataloguing the faults worth injecting. Not every failure is interesting — these five cover the classes that actually change agent behavior.

Step 2: Build a per-test fault fixture

Now the fixture. The key is a single injection point you can configure per test, so a fault is one line in the test body — not a bespoke mock per scenario.

import { test as base } from '@playwright/test';

type Fault = 'error_504' | 'timeout' | 'malformed' | 'empty_result';

export const test = base.extend<{ injectFault: (tool: string, fault: Fault) => Promise<void> }>({
  injectFault: async ({ page }, use) => {
    await use(async (tool, fault) => {
      // Intercept the agent's outbound tool call by tool name
      await page.route(`**/api/agent/tools/${tool}`, async (route) => {
        if (fault === 'timeout') return; // never fulfill — force the deadline
        if (fault === 'error_504') return route.fulfill({ status: 504 });
        if (fault === 'malformed') return route.fulfill({ status: 200, body: '{"trunc' });
        if (fault === 'empty_result') return route.fulfill({ status: 200, json: {} });
      });
    });
  },
});

Step 3: Assert on the side effect and the narration

With the fixture in place, the recovery test reads like a spec. Notice what it asserts on: the side effect (no shipment was actually booked) and the narration (the reply admits failure). Both must hold — that’s the honest-recovery contract.

import { test } from '../fixtures/tool-fault';
import { expect } from '@playwright/test';

test('agent surfaces failure when reschedule tool 504s', async ({ page, injectFault }) => {
  await injectFault('reschedule_shipment', 'error_504');

  await sendMessage(page, 'Move my delivery to Thursday');

  // 1. No fabricated confirmation — the dangerous bug
  await expect(page.getByRole('status')).not.toContainText(/booked|all set|confirmed/i);
  // 2. Honest acknowledgment the user can act on
  await expect(page.getByRole('status')).toContainText(/couldn.t|try again|unable/i);
  // 3. The side effect never happened
  expect(await getBookedShipments(page)).toHaveLength(0);
});

Why not just unit-test the tool wrapper’s error handling?

Because the tool wrapper isn’t where the decision lives — the model is. You can prove your reschedule_shipment client throws cleanly on a 504 all day long, and the agent will still hallucinate a confirmation, because the model is the component that decides what to do with that thrown error. The whole point of an agentic system is that control flow runs through a probabilistic component, and you can only test a probabilistic component end-to-end, with the real prompt and the real tool result in the loop.

This is chaos engineering applied to the agent stack. The Principles of Chaos Engineering argue you build confidence in a system’s resilience by injecting real-world failures and verifying steady-state behavior holds — and that you do it against the integrated system, not isolated units. Netflix proved it for microservices; the same logic transfers directly to agents, where the “unpredictable dependency” is both the network and the model reacting to it. Injecting the fault at the tool boundary and observing the agent’s full response is the only test that exercises the actual failure path.

There’s a scaling concern worth naming: every injected fault is another LLM call, and a full fault matrix across every tool multiplies your CI token bill fast. Be selective — inject faults only on tools with real side effects (writes, payments, external actions), and keep read-only tools on the happy path. If token cost in CI is already a constraint for you, the same token-budget discipline for AI tests in CI applies here, and failed-tool tests are worth the spend precisely because they catch the bugs that reach customers.

What to watch for when a fault test goes flaky

Fault-injection tests have a failure mode of their own: non-determinism in the recovery itself. A 504 might trigger a retry that succeeds on the second call, so the same test passes or fails depending on how the model felt that run. Don’t fight this with retries in the test runner — that hides the signal. Instead, make the fault sticky (fail every call, not just the first) when you’re testing the give-up path, and make it transient (fail once, then succeed) only when you’re specifically testing retry recovery. Separate those into two tests with two fault modes. When a genuinely flaky agent response does slip through, route it through your CI failure triage flow rather than papering over it with retries: 2.

The one thing to do this week

Pick your single highest-stakes tool — the one that moves money, sends something, or changes a booking. Write one test that makes it return a 504, and assert the agent does not claim success. That’s it. One test, one tool, the negative assertion. If it passes, you’ve earned real confidence in your worst-case path. If it fails — and on a suite that’s only ever seen healthy tools, it very often does — you just found the bug that was going to erode your users’ trust one silent failure at a time, and you found it in CI instead of in a support queue.