Your Grounding Tests Pass. Retrieval Recall Dropped.

Someone swapped the embedding model behind your RAG feature — text-embedding-3-small to something newer, or a re-chunk from 512 to 256 tokens, or a vendor bumped their endpoint. Your grounding suite stayed green. Every answer still cited a retrieved chunk, faithfulness still scored above threshold, and the deploy sailed through. Three weeks later support is drowning in “the assistant can’t find our refund policy anymore” tickets. The retrieval recall collapsed on a whole class of queries, and not one test noticed — because none of them were measuring recall.

This is the blind spot that sits directly underneath a grounding test. Grounding proves the answer was faithful to whatever got retrieved; it says nothing about whether the right documents were retrieved in the first place. A confidently grounded answer built on the wrong three chunks passes every faithfulness check you own. When you change the embedding model, the chunker, or the distance metric, retrieval quality is exactly the property that moves — and it’s the one property your green suite isn’t watching. Here’s how to build a deterministic recall@k gate in Playwright so a silent re-embedding can’t ship past you again.

Why do grounding tests pass while retrieval silently regresses?

Because grounding and recall measure two different relationships, and a regression in one is invisible to the other. Grounding (faithfulness) asks “is every claim in the answer supported by the retrieved context?” Recall asks “did we retrieve the documents that actually contain the answer?” A model can be perfectly faithful to a retrieved set that no longer includes the document it needs — it just answers thinly, or hedges, or grounds itself in a near-miss chunk. Faithfulness stays high; the answer gets worse.

I covered the grounding half of this in why a correct RAG answer can still be hallucinated — attribution and faithfulness are the layers you check after retrieval. This post is the layer underneath: retrieval recall itself. In RAG evaluation, recall@k measures whether the relevant documents appear among the top-k retrieved, and missing a single essential chunk is enough to make the final answer incomplete or wrong.

That last column is the trap. Your grounding suite reads the answer and the retrieved context; it never compares the retrieved set against a labeled ground truth. So when recall drops from 0.94 to 0.70 after a re-index, the grounding tests keep passing on the queries that still work, and the ones that broke degrade into vague-but-faithful answers that no assertion is shaped to catch.

What actually changes recall when you “just” upgrade embeddings?

Almost everything about the retrieval stack is a lever on recall, and the three most common “harmless” changes each move it. Swapping the embedding model, re-chunking, or changing the distance metric are not config tweaks — they re-draw the vector space, and documents that used to land in the top-k fall out of it.

The model swap that ranks higher but retrieves worse

Start with the model swap, because it’s the one that hides behind a higher benchmark number. On the MTEB leaderboard, NV-Embed-v2 posts a 72.31 overall score yet a retrieval score of 62.65 — trailing Gemini Embedding 001’s retrieval score of 67.71 despite a lower headline average. “We moved to the higher-ranked model” can mean you moved to worse retrieval. And the swings are large: domain-adapting an embedding model has been shown to move Recall@1 from 34.3% to 62.8% on a finance corpus — the same magnitude of change, in the wrong direction, is what a bad migration does to you.

Chunking, normalization, and the distance-metric trap

Chunking alone is worth up to a 9% swing in retrieval recall depending on strategy, and the optimal size is query-dependent — fact-heavy lookups favor small chunks, summaries favor overlap. The distance-metric trap is subtler: cosine similarity equals dot product only when vectors are normalized to unit length, so flipping the metric without renormalizing silently changes which neighbors win.

How do you build a labeled retrieval gold set?

You write down, for a fixed list of real queries, which document IDs must appear in the retrieved set — and you freeze it. This is the piece teams skip, and it’s the only thing that turns “retrieval feels fine” into a number a test can block on. The gold set is small, hand-labeled, and version-controlled next to the suite; it is your ground truth, so it can’t drift with the index.

The labeling is a few hours of a senior engineer’s time, once. Compare that to the cost of finding out from a customer that your assistant went blind on the refund policy for a sprint.

How do you assert recall@k in Playwright?

Intercept the retrieval call, read the ranked chunk IDs off it, and compare them to your gold set with a plain function — no model, no judge, no flakiness. This is the same move that makes attribution checks deterministic: Playwright watches the network while it drives the UI, so you capture exactly what the retriever returned before the generator ever sees it.

test('retrieval recall@5 holds for the enterprise-refund query', async ({ page }) => {
  const chat = new ChatPanel(page);
  await chat.goto();

  // Retrieval fires before the model call — capture the ranked result set.
  const retrieval = page.waitForResponse(r =>
    r.url().includes('/api/retrieve') && r.ok());
  await chat.ask('What is the refund window for enterprise plans?');

  const { chunks } = await (await retrieval).json();
  const retrievedIds = chunks.map((c) => c.id); // ranked, best-first
});

Recall@k is a five-line pure function. Because it only compares ID sets, it is as deterministic as any assertion in your suite — the non-determinism of the model never enters it.

// Fraction of must-hit gold docs that land in the top-k retrieved results.
export function recallAtK(retrieved: string[], gold: string[], k: number): number {
  const topK = new Set(retrieved.slice(0, k));
  const hits = gold.filter((id) => topK.has(id)).length;
  return gold.length === 0 ? 1 : hits / gold.length;
}

Now wire the frozen gold set into a hard expect. When recall drops below the floor, the failure message names the query and the specific missing document — so the person triaging sees “we stopped retrieving policy-refund-ent-v3” instead of “the eval score went down.”

import gold from './retrieval-goldset.json'; // frozen: query -> { mustRetrieve, k, floor }
import { recallAtK } from './recall';

  const { mustRetrieve, k, floor } = gold['enterprise-refund'];
  const missing = mustRetrieve.filter((id) => !retrievedIds.slice(0, k).includes(id));
  const recall = recallAtK(retrievedIds, mustRetrieve, k);
  expect(recall,
    `recall@${k} = ${recall.toFixed(2)} < floor ${floor}; missing: ${missing.join(', ')}`)
    .toBeGreaterThanOrEqual(floor);

Where should the recall gate run — PR or nightly?

Both, split by cost — and this is the part that makes recall testing practical instead of aspirational. The deterministic ID-set check is cheap and never flaky, so the per-query recall floor belongs in the blocking PR suite against a recorded index snapshot. The expensive part — re-embedding the whole corpus and measuring aggregate recall across the full gold set — runs nightly.

Add one guardrail above both, because the cheapest bug to catch is the contract mismatch itself. A single deterministic test that asserts the live index was built with the embedding model, dimension, and distance metric you expect will fail instantly on a botched re-embed — before recall even runs.

test('vector index matches the embedding contract it was built with', async ({ request }) => {
  const meta = await (await request.get('/api/index/meta')).json();
  // Model + dimension + distance are one contract; a mismatch means a bad re-index.
  expect(meta.embeddingModel).toBe('text-embedding-3-large');
  expect(meta.dimensions).toBe(3072);
  expect(meta.distance).toBe('cosine'); // dot-product on unnormalized vectors != cosine
});

What I’d do differently

The regression that taught me this shipped at a large financial-services client. A platform team re-chunked the knowledge base from 512 to 256 tokens to “improve precision,” re-embedded overnight, and deployed. Every grounding and faithfulness test passed, because the answers that still worked were still faithful. What nobody measured: recall@5 on our labeled policy queries fell from 0.94 to 0.71. For three weeks the assistant gave confident, well-grounded, subtly incomplete answers on split-across-boundary questions, and the only signal was a slow rise in “it used to know this” tickets. We were paying for a green suite that measured the wrong relationship.

Two things I’d do from day one now. First, label the gold set before the first RAG feature ships, not after the first incident — it’s a few hours that would have turned a three-week outage into a red build. Second, I’d make the index-contract test the very first check in the suite, because the day we finally debugged it, the root cause was a distance metric that had silently flipped to dot product on unnormalized vectors — a one-line assertion would have caught it in seconds instead of days.

Your one concrete next step

Take your single highest-traffic RAG query and give it a gold label today: the two or three document IDs that must be retrieved to answer it. Then write the recall@5 assertion — capture /api/retrieve, run the five-line recallAtK, assert it clears a floor. That’s fifteen lines, it runs in your existing Playwright suite with no judge and no new infrastructure, and it will catch the silent re-embedding regression that your grounding tests are structurally blind to. Add the full nightly re-embed comparison later; ship the deterministic floor first.

If you want more field-tested QA automation breakdowns like this — no fluff, just what actually holds up at enterprise scale — subscribe below. And if you’re formalizing how your team tests AI features, the rest of the AI-powered testing pillar goes deep on grounding, judges, drift, and mocking strategy.