Agentic AI Said the Test Passed. It Didn't.

The agent’s run log was up on the screen at the incident review, and it read beautifully:

✓ Navigated to /checkout
✓ Applied promo code SPRING24
✓ Order confirmation displayed — order #A-441902
✓ Read totals from receipt: Subtotal $31.54 · Tax $4.10 · Total $35.64
VERDICT: PASS
Checkout completed end to end. Totals rendered correctly.

Read line 4 again. The agent printed the wrong number and then called it correct. Those were zero-rated safety boots. The tax should have been $0.00. We shipped it on a Tuesday, and six days later a finance analyst reconciling remitted sales tax found it — roughly 1,900 orders, every one of them a remittance correction rather than a refund. Not the suite, not the agent, not us.

Now look at what that failure produced on its way through our pipeline: nothing. No red run, no rerun, no flake, no retry, no dashboard row. The industry is busy treating agentic AI test pass/fail reliability as an accuracy dial — get the agent good enough and you can let it decide. That framing misses what the failure actually is.

Triage assumes there’s a failure to explain. A flaky judge at least disagrees with itself loudly enough to annoy you into fixing it. But an agent that owns the verdict on a page that looks fine emits no event at all, and no pipeline can retry, alert on, or triage an event that was never emitted. That’s why the verdict is the one job an agent can’t have. Not because the model is stupid — because a test that can never fail is invisible to every safety net you own.

The question is live: the Ministry of Testing thread on replacing manual testing with agentic AI runs fourteen posts deep without a single practitioner reporting they’ve actually done it. The sticking point every time is trusting the agent to call pass or fail.

Why is agentic AI pass/fail reliability the wrong thing to measure?

Because reliability is scored against ground truth, and on a computed business value the agent has no ground truth to be scored against. It has an artifact — a rendered page, a DOM snapshot, an HTTP 200 — and it must infer correctness from it. So it grades the only thing it can actually evaluate: plausibility. A $4.10 tax line on a $31.54 cart is extremely plausible. It is, in fact, exactly what that cart would owe if the boots weren’t zero-rated.

That’s the bug twice over. The pricing service answered a different question than the one it was asked, and so did the agent.

Notice what makes this worse than a model copying a wrong number out of an implementation — the generation-time trap I’ve written about in AI-generated code and who writes the oracle. Here the correct value was zero. There was no number anywhere to copy, and zero is the one answer a plausibility judge will never reach for, because zero doesn’t look like a tax.

None of this is new, and that’s the point. Barr, Harman, McMinn, Shahbaz and Yoo named it in 2015: given an input, distinguishing correct behaviour from plausible-looking incorrect behaviour is the test oracle problem, which they called “a significant bottleneck that inhibits greater test automation.” An LLM oracle does not lift that bottleneck. It conceals it, because it always produces an answer.

Isn’t a wrong verdict just another flaky test?

No, and the difference is the entire blast radius. A flaky judge is noisy — it contradicts itself, you catch it on a rerun, and it irritates you into action. A miscalibrated oracle on a deterministic path is stable. It returns the same confident PASS on every run, forever, and nothing in your pipeline is built to notice an assertion that never fires.

I’ve written before about the judge that disagrees with itself between runs — position bias, swap inconsistency, the works. That post is about variance. This is the opposite failure: bias toward pass. And the 2026 production data on it is genuinely alarming.

A study of a deployed LLM judge on multi-turn transaction agents (Catching One in Five) found it caught 2 of 9 human-confirmed systematic defects. In one closed batch its operational gate flagged zero of 100 rounds, while human reviewers confirmed 23 distinct defects in the same set. Then the finding that should end the debate: in 113 of 114 rounds where the judge’s own notes described a real cart or confirm-gate defect, the note was filed under brand voice — and not one of those defects ever reached a failing verdict. The authors put it better than I would: “The failure is one of routing, not perception.”

That’s my agent, exactly. It read Tax $4.10 out loud, wrote it into the log, and filed it under “rendered correctly.” It perceived the defect. It had no rubric axis that said this number must equal a value that came from somewhere other than this page, so the observation had nowhere to go.

Nor is this something a better model fixes. In one 45-task audit, agents reported 45 of 45 complete while only 26 passed the held-out tests — and two frontier models from different labs false-passed the identical 19 tasks. That’s not a defect you upgrade away. That’s the objective function.

Which gives the rule that actually matters:

A verdict you cannot falsify isn’t a test result. It’s a status report with a nice font.

Ask it of any agent-driven test you own: what would have to be different for this to come back red? If the answer is “the page would have to error out,” you don’t have a test on that flow. You have a smoke check wearing a verdict.

Where does the deterministic assertion have to live?

On every value a customer could dispute in a chargeback, and it has to arrive from a source your implementation never touched. Money, tax, entitlement, quantity, permission, state transition. If the wrongness is expressible as a number, the oracle must be a number. A natural-language verdict judging an integer is a precision downgrade, and you pay for it in remittance corrections.

Assert on what the server computed, not on what the agent saw

The DOM is the last place the value lands and the worst place to judge it. Catch the quote response and assert on the payload in integer cents — no floats, no currency parsing, no pixels:

import { test, expect } from '@playwright/test';
import { TAX_GOLDEN } from '../fixtures/tax-golden';

for (const cart of TAX_GOLDEN) {
  test(`${cart.name} is charged ${cart.taxCents} cents of tax`, async ({ page }) => {
    await page.goto(`/cart/${cart.name}`);
    const quote = page.waitForResponse('**/api/checkout/quote');
    await page.getByRole('button', { name: 'Place order' }).click();

    // Server truth: integers, sourced from the API, not the receipt.
    const body = await (await quote).json();
    expect(body.taxCents).toBe(cart.taxCents);
    expect(body.grandTotalCents).toBe(cart.totalCents);

    // Customer truth: what's printed on the page they're charged from.
    const dollars = (cart.totalCents / 100).toFixed(2);
    await expect(page.getByTestId('order-total')).toHaveText(`$${dollars}`);
  });
}

An agent can rewrite a locator. It cannot negotiate with toBe(0).

Never let the test recompute the business logic

This is the trap teams fall into five minutes after reading the section above. You write expect(total).toBe(subtotal * 1.13) — and you’ve just reimplemented the pricing rule inside the test, which means the test now agrees with the bug whenever the bug is a misreading of the rule. That’s a mirror, not an oracle. Expected values have to come from outside the system that produces them:

// Sourced from the tax matrix Finance signs off on each quarter.
// NOT derived from the pricing service, NOT computed in this file,
// NOT captured from a run that happened to be green.
export const TAX_GOLDEN = [
  { name: 'zero-rated-safety-boots', subtotalCents: 3154, taxCents: 0,   totalCents: 3154 },
  { name: 'standard-hst-ontario',    subtotalCents: 3154, taxCents: 410, totalCents: 3564 },
  { name: 'mixed-exempt-taxable',    subtotalCents: 9954, taxCents: 884, totalCents: 10838 },
] as const;

Three lines of committed data, reviewed by someone who reads tax law for a living. Row one is the test that would have held the release. Row two is the number the agent saw and blessed.

The agent is still allowed to do almost everything else

I’m not arguing against agents in the loop. Let them explore the app, draft the plan, generate the spec, and triage the failures they didn’t cause. Every one of those outputs is reviewable — a Markdown plan, a code diff, a ranked hypothesis — and every one lands in front of a human before it can hurt anyone. A verdict lands in a merge gate, and a merge gate has no eyes.

Doesn’t that mean the agent never solved the oracle problem?

Correct, and the industry should say so out loud. The consensus prescription — the agent navigates, deterministic assertions judge — quietly concedes the whole game. If a human still writes the assertion and sources the expected value, the agent solved locator brittleness. It did not solve the oracle problem. That’s a real and useful win; it is not autonomous QA, and it should stop being sold as one.

Be careful where you let an agent near the assertion at all. Playwright’s healer looks like a maintenance tool, but its shipped agent definition instructs it to remediate by “fixing assertions and expected values” and to “do the most reasonable thing possible to pass the test” — with test.fixme() as its fallback, which the official docs describe as “a skipped test if the healer believes that functionality is broken.” Run it in shadow mode and read every diff before it touches a spec on the money path.

What I’d do differently

I’d make every agent-driven run emit the set of deterministic assertions it actually evaluated, and fail the run when that set comes back empty on a flow tagged @money. An agent that can’t name its falsifier doesn’t get to vote. We had built an artifact that structurally could not fail, and nobody noticed for two sprints, because nothing was ever red.

I’d also measure the thing nobody measures: inject known defects into a branch — a dropped exemption flag, an off-by-one tier boundary, a swapped currency unit — and count how many the agent waves through. That’s your real false-pass rate on your own app, and it’s the same mutation-testing move that proves an assertion asserts anything at all.

Your one concrete next step, this week: take your highest-value flow and commit a single file — a three-row golden table of expected values, sourced from whoever owns the spec rather than from your code, your app, or a run that happened to be green. Then delete every assertion on that flow that doesn’t compare against a row in it. If that leaves you with zero assertions, you’ve just found out what your agent has been vouching for.

More on where agents earn their place in a suite — and where they quietly cost you — in the AI-powered testing archive.