Your AI Agent's Tool Calls Come Out of Order

You did the hard part. You stopped asserting on the chat bubble and started asserting on the tool calls your agent actually makes. The test captures search_inventory, then check_price, then add_to_cart, and it’s green. Then it flakes — one run in six — because this time the agent fired check_price before search_inventory, your toEqual on the ordered array blew up, and nothing about the product was actually wrong. You’ve moved the assertion to the right boundary and inherited a new flake: the order.

Here’s the uncomfortable part. That flake is not a bug in your agent. Parallel tool calling means the order is allowed to change, by design, and your assertion is the thing making a false promise. The fix isn’t to pin the order — it’s to stop asserting on order you were never guaranteed, and to assert precisely the order you actually depend on.

Why do parallel tool calls arrive in a different order?

Because the model emits them as an unordered batch and the API explicitly refuses to promise a sequence. When Claude decides to act, it can return several tool_use blocks in a single turn, and Anthropic’s own guidance states plainly that the API doesn’t prescribe an execution order — you may run those calls concurrently, sequentially, or in any combination. OpenAI’s parallel function calling works the same way: the model returns a list of tool_calls, and nowhere does the documentation guarantee that list is stable run to run.

So the sequence your test captured on Tuesday was one valid serialization of an unordered set. Your backend probably runs those calls with Promise.all or asyncio.gather for latency, which means even the arrival order at your network layer is a race. The DOM, the network log, the trace — all of them record whatever order won the race that run. Pin an assertion to it and you’ve hard-coded a coin flip.

This matters now because agents that act are no longer a demo. In LangChain’s State of Agent Engineering survey, 57% of surveyed teams already run agents in production, and the multi-tool agent — the one that fans out three or four calls per turn — is the common shape. The more tools you give an agent, the more parallel batches it emits, and the more your ordered assertions turn into a slow-motion flake generator.

Why is an ordered assertion the wrong contract?

Because you’re testing an implementation detail the model is free to change, not the behavior you actually care about. There are two different questions hiding inside “did the agent call the right tools,” and ordered equality answers neither cleanly:

• Did it call the right set of tools with the right arguments? (Almost always what you mean.)
• Did it respect the one ordering that genuinely matters? (Occasionally what you mean — and when it does, it matters a lot.)

Ordered toEqual on the whole array conflates these. It fails when a harmless permutation happens and it can pass when a genuinely dangerous reordering happens, as long as the array positions line up. It’s both too strict and too weak. The Berkeley Function-Calling Leaderboard exists because models make real, measurable errors in which function they call and what arguments they pass — those are the failures worth catching. Argument-in-cents-not-dollars is a real bug. check_price landing 40ms before search_inventory in an independent batch is not.

What does the flaky version look like?

Here’s the pattern that ships. It captures tool calls off the network boundary — the right boundary — and then makes exactly one mistake: it compares the whole sequence with ordered equality.

// Collect every tool call the agent's backend emits
async function captureToolCalls(page) {
  const calls: { name: string; args: any }[] = [];
  await page.route('**/api/agent/step', async (route) => {
    const body = route.request().postDataJSON();
    if (body?.tool) calls.push({ name: body.tool, args: body.arguments });
    await route.continue();
  });
  return calls;
}

test('agent adds the item to cart', async ({ page }) => {
  const calls = await captureToolCalls(page);
  await runAgent(page, 'Buy one of the blue widgets');
  // ❌ Order-sensitive: flakes when the parallel batch permutes
  expect(calls).toEqual([
    { name: 'search_inventory', args: { q: 'blue widget' } },
    { name: 'check_price',      args: { sku: 'BW-01' } },
    { name: 'add_to_cart',      args: { sku: 'BW-01', qty: 1 } },
  ]);
});

The capture logic is fine. Line 16’s toEqual on an ordered array is the flake. search_inventory and check_price are independent — the model can and will emit them in either order — so one run in several, the array positions swap and CI goes red for a non-reason. Your team learns to re-run it, and a re-run culture is exactly how a green check mark stops meaning anything.

How do you assert the set instead of the sequence?

Compare tool calls as an unordered set of { name, args } objects. You still catch a missing call, a wrong tool, and a bad argument — the failures that matter — and you stop caring about serialization order you were never promised.

import { expect } from '@playwright/test';

// Order-independent: every expected call must appear, with matching args
export function expectToolSet(actual, expected) {
  const norm = (c) => JSON.stringify({ name: c.name, args: c.args });
  const actualSet = actual.map(norm).sort();
  const expectedSet = expected.map(norm).sort();
  expect(actualSet).toEqual(expectedSet);
}

test('agent adds the item to cart', async ({ page }) => {
  const calls = await captureToolCalls(page);
  await runAgent(page, 'Buy one of the blue widgets');
  // ✅ Set-based: permutations pass, wrong/missing/malformed calls fail
  expectToolSet(calls, [
    { name: 'search_inventory', args: { q: 'blue widget' } },
    { name: 'check_price',      args: { sku: 'BW-01' } },
    { name: 'add_to_cart',      args: { sku: 'BW-01', qty: 1 } },
  ]);
});

But some orders really do matter — how do you keep those?

This is the part people miss when they hear “stop asserting order.” Some sequences are load-bearing: authenticate must precede issue_refund; create_record must precede attach_file. Those aren’t parallel-batch noise — they’re causal dependencies, and if the agent violates them you have a genuine, shippable bug. The move isn’t to drop all ordering. It’s to assert the partial order — only the edges that are real — and let everything else float.

// Assert only the edges that are real dependencies
export function expectHappensBefore(calls, before: string, after: string) {
  const i = calls.findIndex((c) => c.name === before);
  const j = calls.findIndex((c) => c.name === after);
  expect(i, `${before} must be called`).toBeGreaterThanOrEqual(0);
  expect(j, `${after} must be called`).toBeGreaterThanOrEqual(0);
  expect(i, `${before} must happen before ${after}`).toBeLessThan(j);
}

test('agent refunds only after authenticating', async ({ page }) => {
  const calls = await captureToolCalls(page);
  await runAgent(page, 'Refund my last order');
  expectHappensBefore(calls, 'authenticate', 'issue_refund');
});

You’ve now encoded the actual contract: this exact edge is mandatory, everything else is free. A permutation of independent calls passes; an agent that refunds before authenticating fails every single time, not one time in six. That’s a partial order — a DAG of the dependencies that matter — and it’s the honest description of what a correct agent does.

What about duplicate and retried calls?

Parallel batches surface a second-order trap: the same tool called twice. An agent that retries check_price after a transient error, or fans out check_price per SKU, produces multiple identical-looking calls. A naive set that dedupes will hide a real over-calling bug — an agent hammering a paid pricing API twice per request is a cost incident waiting to happen. Keep multiplicity in your comparison (the .sort() on a plain array above preserves duplicates rather than collapsing them), and when a call should be idempotent, assert the count explicitly:

const priceCalls = calls.filter((c) => c.name === 'check_price');
expect(priceCalls, 'price should be checked exactly once').toHaveLength(1);

A war story on getting this wrong

At a retail client last year, the agent-driven checkout suite had one test everybody re-ran. Green on the second try, always. For two months the team treated it as environmental — “CI’s flaky today.” I pulled the traces from the last 30 failures and every single one was the same: search_inventory and check_price swapping places in a parallel batch, tripping an ordered toEqual. Zero of those 30 failures were real defects. We’d trained an entire squad to ignore a red build on the checkout path — the exact path where a real refund-before-auth bug would have looked identical to the noise.

The rewrite and the numbers

We rewrote it: one expectToolSet baseline, plus a single expectHappensBefore('reserve_stock', 'add_to_cart') because reserving before adding was a real inventory rule. Flake rate went from roughly one in six to zero over the next 400 CI runs, and — this is the point — the suite got stricter, not looser. The set check caught a qty argument regression the following sprint that the old ordered assertion had been positionally blind to.

What I’d do differently

I’d have looked at the failure traces on day one instead of trusting the “flaky CI” story for two months. Thirty red builds with an identical, benign cause is a five-minute diagnosis if you actually read them — and the same discipline that catches a race in a streaming AI response applies here. Flake is a hypothesis to investigate, not a weather condition to wait out. I’d also have written the partial-order helper as shared infrastructure the first time, not the third, so every agent test in the org inherited the right default.

The one thing to do today

Grep your agent test suite for toEqual and toStrictEqual applied to a captured array of tool calls. Every one of those is an order-sensitive assertion over a batch the model is free to reorder. Replace each with a set-based check plus an explicit happens-before for the one or two dependencies that are genuinely causal. You’ll delete a class of flake and, more often than not, tighten the test at the same time — because you’ll finally be asserting behavior instead of serialization.

If you want the deeper foundation this builds on, start with asserting on tool calls instead of the reply, then treat this post as the parallel-batch sequel.