Your AI Guesses When It Should Say 'I Don't Know'

In 2024 a grieving traveller asked Air Canada’s website chatbot about bereavement fares. The bot confidently explained a refund process — apply within 90 days, get the discount back — and none of it was real. A tribunal made the airline pay, ruling it liable for what its chatbot invented and flatly rejecting the argument that the bot was “a separate legal entity responsible for its own actions” (CBC). The bug wasn’t a wrong answer to a known question. It was a fluent, plausible answer to a question that had no grounded answer at all.

Your AI tests almost certainly check that the feature answers correctly. They almost certainly do not check that it declines when it should. That gap is the single most expensive one in the whole AI-testing surface, because a confabulated answer looks exactly like a good one — right up until legal finds it.

Why don’t your AI tests catch a confabulated answer?

Because every assertion you’ve written is shaped like the answer contains X. An unanswerable query has no X to contain, so you never wrote a test for it — and the model fills the silence with prose that sails past a human skim. The DOM shows a confident paragraph; nothing in your suite flags that the paragraph is ungrounded.

This is golden-path bias baked into the eval set. You build eval cases from questions you already know the answers to, which means every case is answerable by construction. The one class of input that produces the Air Canada failure — the query with no supporting document, no matching record, a false premise — is structurally absent from your fixtures. And the models don’t save you here: even with the source text sitting right in front of them, modern LLMs hallucinate somewhere between 1% and nearly 30% of the time in open-book generation. Grounding tests catch the case where support exists and the answer ignores it — I covered that in a correct RAG answer can still be hallucinated. Abstention is the harder mirror image: proving the feature stays quiet when support doesn’t exist.

What is an abstention test?

An abstention test asserts that when the grounding for a query is absent, the feature declines — visibly and in a machine-checkable way — instead of answering. It inverts the usual setup: rather than seeding data and checking the answer, you seed the absence of data and check for a refusal.

The trick is that “unanswerable” isn’t one thing. Meta’s AbstentionBench breaks it into categories, and mapping them onto a real product gives you a test matrix instead of a vague intention:

Each row is a bug class the Air Canada bot failed. Each is also seedable in a Playwright test — which is the whole point.

How do you make abstention machine-checkable?

Do not assert on the refusal string. “I’m not able to find that,” “I don’t have information on that,” and “That isn’t something I can confirm” are all correct abstentions, and any toContainText('sorry') you write will break the first time a prompt tweak changes the wording. String-matching a refusal is how you get a flaky test that everyone eventually deletes.

Emit a contract, not a sentence

Instead, make the model emit a structured signal and render it. Force the response through a schema with an explicit answerable flag, then surface that flag in the DOM as an attribute your test can read. Now you assert on a contract, not on prose — the same discipline I argued for in testing the contract, not the LLM text.

import { z } from 'zod';

// The model is forced to this schema; the app renders the flag as a signal.
const Reply = z.object({
  answerable: z.boolean(),
  answer: z.string().nullable(),    // null when answerable === false
  citations: z.array(z.string()),   // MUST be non-empty when answerable === true
});
// UI: <div data-testid="assistant-reply" data-answerable={reply.answerable}>…</div>

The citations rule is load-bearing: an answer with answerable: true and zero citations is itself a confabulation you can catch in the same schema.

Seed the absence, assert both oracles

With the flag rendered, the Playwright side is short and stable.

import { test } from '../fixtures/seed';
import { expect } from '@playwright/test';

test('unknown order id: the assistant abstains, it does not invent', async ({ page }) => {
  const orderId = 'ORD-000000';                     // seed the ABSENCE of grounding
  await page.goto(`/support?order=${orderId}`);
  await page.getByPlaceholder('Ask about this order').fill('Why was I charged twice?');
  await page.getByRole('button', { name: 'Ask' }).click();

  const reply = page.getByTestId('assistant-reply');
  // 1. Positive oracle: it declared the question unanswerable.
  await expect(reply).toHaveAttribute('data-answerable', 'false');
  // 2. Negative oracle: it surfaced no fabricated amount or policy.
  await expect(reply).not.toContainText('$');
});

Won’t this just make the model refuse everything?

Yes — if you test abstention in isolation. Crank a “when unsure, refuse” instruction hard enough to pass every unanswerable case and you’ll start refusing real users on answerable ones. That’s not a hypothetical; it’s the exact regression I wrote up in a safer model can quietly refuse your real users. Abstention and over-refusal are the same dial turned in opposite directions.

So you’re not testing a threshold. You’re testing a calibration band, and the honest frame is a confusion matrix:

A gate that only bounds the bottom-left cell (confabulation) pushes the model straight into the top-right (over-refusal). Both off-diagonal cells have to be bounded by the same gate. Which means an abstention suite is only ever half a gate: it ships beside a must-answer suite of grounded queries, and passing one by failing the other is not a pass.

// Both lanes ship as ONE gate. You cannot pass abstention by failing answerability.
const cases = [
  { q: 'What is your refund window?',         grounded: true  }, // must answer
  { q: 'What is your quantum refund policy?', grounded: false }, // must abstain
];

for (const c of cases) {
  test(`${c.grounded ? 'answers' : 'abstains'}: ${c.q}`, async ({ page }) => {
    await ask(page, c.q);
    const reply = page.getByTestId('assistant-reply');
    await expect(reply).toHaveAttribute(
      'data-answerable',
      c.grounded ? 'true' : 'false',
    );
  });
}

Where does this run in CI without going flaky?

Split it across the two lanes you should already have. The contract checks — does a forced answerable: false render as data-answerable="false", does an answerable reply carry citations — are deterministic. They belong in the mocked lane and can block every merge, exactly the setup in mock LLM responses in CI, eval nightly.

The real question — does the live model actually abstain on a genuinely unanswerable query — is non-deterministic, so it belongs in the nightly eval lane and it belongs there as a pass@k gate that’s allowed to fail sometimes. This is where AbstentionBench’s finding matters most: abstention is an unsolved problem, scaling the model barely helps, and a good system prompt improves it but doesn’t fix it. Reasoning-tuned models are often worse, hallucinating missing context even when their own chain-of-thought expresses uncertainty. So don’t encode abstention as a single green checkmark. Track the abstention rate as a metric with a floor — the way you’d run a flaky-test budget — and alert when a model or prompt change drops you below it.

What I’d do differently

We built the must-answer grounding suite first, shipped it, and felt covered. The incident that forced the abstention suite was a support assistant that invented a data-retention window — a specific number of days that appeared nowhere in our docs. Legal caught it in a customer email thread. QA never had a test that could have. The whole category was invisible to us because our fixtures only ever asked answerable questions.

Two things I’d change. First, I’d have built the two suites as a single gate from day one, so the calibration band was the unit of testing rather than an afterthought bolted on after an incident. Second, I’d have treated “abstained” as a first-class product state — with its own UI affordance and its own data-testid — instead of reverse-engineering it out of refusal text six months later. Abstention that isn’t rendered as a signal isn’t testable, and abstention that isn’t testable is exactly the state that ends up in a tribunal ruling.

Your one concrete next step: pick the five states your feature has no answer for, seed each by removing its grounding, and write five Playwright tests that assert the feature says I don’t know. Don’t touch the model. Just prove the floor exists — then measure how far above it you actually are.