Your AI Agent Retries and Charges the Card Twice

Your payments agent called charge_card, the backend committed the charge, and then the gateway dropped the response on the floor. The agent saw a timeout, did the reasonable thing, and called charge_card again. Now the customer has two authorizations, your dispute queue has a new entry, and your eval suite is still green because every test you wrote assumed the tool either succeeds or fails — never both. The model didn’t hallucinate. It didn’t pick the wrong tool. It executed a correct side effect twice because nothing in your system enforces exactly-once.

This is the class of bug that survives contact with a “passing” agent. It doesn’t show up in golden-set accuracy, it doesn’t show up in a trajectory test, and it doesn’t show up until a real network blip lands in the two-hundred-millisecond window between commit and acknowledgment. I’ve watched a 0.3% gateway timeout rate on an anonymized fintech’s payments agent turn into 40-plus duplicate authorizations in its first week live — not because the model was bad, but because two independent retry layers both fired against a non-idempotent tool. Below is why that happens, why the obvious fix makes it worse, and the Playwright test that actually catches it.

Why does an AI agent duplicate a side effect it already completed?

Because a lost response is indistinguishable from a failed request, and an agent has two layers that retry on that ambiguity. The tool ran, the row committed, the money moved — but the caller never got the 200, so from the caller’s point of view nothing happened. Retrying is the correct behavior for a lost request. It’s a catastrophe for a lost response. The system can’t tell them apart from the outside.

The first retry layer is your model SDK. Both the OpenAI Python SDK and the Anthropic SDK automatically retry certain errors — connection errors, timeouts, 5xx — twice by default, before your code ever sees the failure. The second layer is the agent loop itself: when a tool result comes back as an error, a well-behaved agent is supposed to retry or fall back, which is exactly the recovery behavior you want it to have. Stack the two and a single dropped acknowledgment can produce three executions of a side effect the user asked for once.

The uncomfortable part for anyone who came up through distributed systems: this is just the two generals problem wearing a language-model costume. Exactly-once delivery is impossible over an unreliable channel; the best you get is at-least-once delivery plus idempotent handling. Agents didn’t repeal that theorem. They just added a second, chattier client that retries with even less supervision than your old microservice did.

Why doesn’t a fresh idempotency key fix it?

Because most teams generate the key in the wrong place. The instinct is right — Stripe’s idempotency keys are the textbook fix: send a unique key with a mutating request, and the server returns the saved result of the first call instead of running it again (Stripe caches that result for 24 hours). The failure is subtler. Teams mint the key inside the tool handler, per invocation — so the retry generates a brand-new key, the server sees a request it’s never seen, and it happily charges again. You added idempotency infrastructure and got exactly zero idempotency.

Derive the key from the tool call, not the handler

The key has to be stable across every retry of the same logical intent, which means it can’t be born at call time. Derive it from something that survives the retry: the model’s tool_call.id, which is identical when the SDK replays the same response, hashed together with the arguments so a genuinely new intent gets a genuinely new key.

import { createHash } from 'node:crypto';

// Stable across SDK + agent retries of the SAME tool call.
// tool_call.id is constant when the model response is replayed;
// hashing the args guards against id reuse across turns.
export function idempotencyKey(toolCallId: string, args: unknown): string {
  const canonical = JSON.stringify(args, Object.keys(args as object).sort());
  return createHash('sha256')
    .update(`${toolCallId}:${canonical}`)
    .digest('hex')
    .slice(0, 32);
}

Guard the effect, don’t just store the key

Then the handler has to actually use it as a guard, not just log it. First writer wins; every later caller with the same key reads back the stored result without re-running the effect.

export async function chargeCard(args: ChargeArgs, key: string, store: KeyStore) {
  const seen = await store.get(key);
  if (seen) return seen.result; // replay the first outcome, don't re-charge

  const result = await gateway.charge(args); // the real, non-repeatable effect
  await store.put(key, { result, at: Date.now() }); // commit BEFORE returning
  return result;
}

How do you test exactly-once in Playwright?

You inject the one failure real chaos testing usually misses: a success on the server followed by a lost response to the client. Killing the request before it lands proves nothing — the effect never ran. The bug only exists when the effect commits and the acknowledgment vanishes, so that’s the fault your test has to manufacture. Mock the model so the tool call is deterministic, point the agent at a real test backend that counts executions, and use route interception to let the first request through to the handler and then swallow its response.

import { test, expect } from '@playwright/test';
import { startCountingBackend } from './fixtures/counting-backend';

test('a lost tool response does not double-charge', async ({ page }) => {
  const backend = await startCountingBackend(); // increments on real commit
  let attempts = 0;

  // Let the request reach the handler; drop the RESPONSE on attempt 1.
  await page.route('**/tools/charge_card', async (route) => {
    attempts += 1;
    if (attempts === 1) {
      await backend.commitCharge(route.request().postDataJSON()); // effect runs
      await route.abort('connectionaborted'); // ...but the ack never arrives
    } else {
      await route.continue(); // retry hits the idempotent handler
    }
  });

  await runAgent(page, 'Pay invoice #4471');

  expect(attempts).toBeGreaterThan(1);        // a retry did happen
  expect(backend.chargeCount).toBe(1);        // ...and it was absorbed
});

The two assertions matter equally. attempts > 1 proves the retry actually fired — otherwise you’re testing a code path that never executes and calling it green, the same trap as a fault test that quietly stops injecting faults. chargeCount === 1 proves the idempotency guard absorbed it. Assert only the second and a broken key generator still passes on the day the retry doesn’t happen to fire.

Where should this test run, and what should it gate?

On the pull-request path, mocked and deterministic — this is cheap, it’s fast, and it has nothing to do with model quality, so it does not belong in your expensive nightly eval. Treat exactly-once as a contract every mutating tool must satisfy, the same way you’d assert an agent’s tool calls rather than its prose, and run it as a parameterized suite across every tool with a side effect: charges, refunds, shipment creation, email sends, ticket writes. Any tool that can’t survive a replayed call is a production incident with a countdown timer.

One caveat worth its own test: idempotency keys expire. Stripe’s window is 24 hours; if your agent can legitimately resume a task a day later, a retry past the TTL will re-execute. And a retry storm under provider rate limiting can push attempts outside a shorter in-memory cache. Pick a TTL that outlives your longest legitimate retry window, and test the boundary — a key that expires between the effect and the retry is a double-charge with a delay.

The one thing to do today

Pick your single highest-blast-radius tool — the one that moves money, creates an order, or sends something a customer sees — and write the lost-response test for it before you write another eval case. You don’t need the whole framework: mock the model to call that one tool, commit the effect, abort the response, retry, and assert the side effect ran exactly once. If it fails, you just found a duplicate-charge bug before your customers did. If it passes, you’ve turned “we think it’s idempotent” into a check that runs on every pull request — which is the only version of that sentence a senior SDET is allowed to say out loud.