Your AI Feature Works at 5 Docs and Breaks at 50

Every fixture in your AI test suite has three or four documents in it, because that’s what fits comfortably on a screen when you’re writing the seed data. Every real record in production has forty. Your suite is green, your feature is degrading, and no assertion you own can see the difference — because the failure mode isn’t “wrong output,” it’s “quietly worse output as the input grows.”

The fix isn’t a bigger fixture. It’s a context ladder: run the same gold-set question at 1, 5, 15, 40, and 80 documents, measure the score at each rung, and gate on how much you retain relative to the 1-document baseline. That gives you a number your team doesn’t currently have — your feature’s effective context length — and lets you assert that your production p95 volume sits safely below it.

Why does an AI feature get worse when you give it more context?

Two independent things break at volume, and from the DOM they look identical. Your prompt assembler silently drops documents to stay inside a token budget, and the model’s attention degrades on the documents that survive. One is a bug in your code. The other is a property of every model you will ever ship on.

The model half is well measured now and worse than the marketing suggests. NoLiMa (ICML 2025) evaluated 13 LLMs that all claim 128K+ context; at just 32K, 11 of them dropped below 50% of their own short-context baseline, and even GPT-4o fell from 99.3% to 69.7%. NVIDIA’s RULER puts effective context — the longest window where a model still scores 85% — at roughly 50–65% of advertised capacity for most models. Chroma’s Context Rot report reproduced the same shape across 18 models including GPT-4.1, Claude 4, and Gemini 2.5: performance does not decay smoothly, and it is not uniform across the window.

The oldest of these results is still the most useful one for test design. Lost in the Middle (Liu et al., TACL) showed a U-shaped curve: models answer well when the relevant document sits at the very start or the very end of the context and measurably worse when it sits in the middle. That means document count isn’t the only variable — position is. A test that always appends the needle last will pass forever and tell you nothing.

What actually broke in production

A large Canadian insurer, claims-summary assistant. UAT fixtures were built from four-document claims — an FNOL form, two photos, one adjuster note, about 6K tokens. Production claims at p95 carried 63 documents and roughly 180K tokens.

The assembler had a 120K budget and a “keep the newest until we fit” trim, which is the default shape of trim_messages and of every hand-rolled sliding window I’ve reviewed. The policy-exclusions document is attached at underwriting time, so it is always the oldest document on the claim. It was the first thing dropped, on every large claim, for three weeks.

The summaries were fluent. They were also confidently wrong about coverage, which in claims is the only sentence that matters. Every Playwright test passed the entire time, because every Playwright test asserted that a summary rendered.

How do you tell whether it’s your code or the model?

Run the truncation test first. It’s deterministic, it’s cheap, and if it fails you’ve found a plain bug that has nothing to do with AI — which means you can fix it today instead of scheduling an eval project.

The change on the server is small. Stop returning a bare array.

// BEFORE — drops silently, returns nothing you can assert on.
export function assembleContext(docs: Doc[], budget = 120_000): Doc[] {
  const kept: Doc[] = [];
  let total = 0;
  for (const doc of [...docs].reverse()) {   // newest first
    if (total + doc.tokens > budget) break;  // ← every remaining doc vanishes here
    kept.push(doc);
    total += doc.tokens;
  }
  return kept.reverse();
}

// AFTER — same policy, but it tells you what it did.
export function assembleContext(docs: Doc[], budget = 120_000): ContextReceipt {
  // ...same loop, plus: dropped ids, kept ids, total tokens, budget, policy name
}

Surface that receipt on the response — a x-context-receipt header behind the same environment gate you use for any other test-only affordance — and the assertion becomes ordinary Playwright.

import { test, expect } from '@playwright/test';
import { seedClaim, receiptFor } from '../helpers/claims';

test('nothing is dropped at production p95 document volume', async ({ page }) => {
  // 63 documents, not the 4-document fixture everyone writes by hand.
  const claim = await seedClaim({ documents: 63, include: ['policy-exclusions'] });

  const receipt = receiptFor(page, '/api/claims/summary');
  await page.goto(`/claims/${claim.id}`);
  await expect(page.getByTestId('summary')).toBeVisible();

  const { keptIds, droppedIds, tokens, budget } = await receipt;

  expect(tokens).toBeLessThanOrEqual(budget);        // we stayed in budget
  expect(droppedIds).toEqual([]);                    // and did it without discarding input
  expect(keptIds).toContain('policy-exclusions');    // the canary that broke us
});

The canary assertion is the one worth stealing. Pick the single document your domain cannot produce a correct answer without — the exclusions page, the amendment, the latest lab result — and assert it survived assembly. It costs one line and it is the only assertion in the file that encodes domain knowledge.

This whole test belongs in your mocked PR lane. It never calls a model, so it’s free, fast, and deterministic — exactly the split I argued for in mocking LLM responses in CI and evaluating nightly.

How do you measure your feature’s effective context length?

Borrow RULER’s definition and apply it to your app instead of to a model. RULER calls a model’s effective context the longest window where it still scores 85% on the benchmark. Yours is the largest document count where your gold-set score stays within 85% of its 1-document baseline.

That reframing matters because it turns an unbounded question (“is our AI good enough?”) into a bounded one you can put in a pipeline: does our effective context still exceed our production p95? Both numbers are knowable. One comes from a test, the other from a SELECT percentile_cont(0.95).

The ladder in Playwright

Playwright’s parameterized tests make the ladder about fifteen lines:

import { test, expect } from '@playwright/test';
import { seedClaim, askAndGrade } from '../helpers/claims';

const LADDER = [1, 5, 15, 40, 80];
const RETENTION_FLOOR = 0.85;   // RULER's threshold, applied to our own baseline

let baseline: number;

test.describe.serial('context ladder', () => {
  test('baseline at 1 document', async ({ page }) => {
    baseline = await askAndGrade(page, await seedClaim({ documents: 1 }));
    expect(baseline).toBeGreaterThan(0.9);   // if this fails, the ladder is meaningless
  });

  for (const documents of LADDER.slice(1)) {
    test(`retains quality at ${documents} documents`, async ({ page }) => {
      const claim = await seedClaim({ documents, needle: 'policy-exclusions' });
      const score = await askAndGrade(page, claim);

      test.info().annotations.push({ type: 'rung', description: `${documents}:${score}` });
      expect(score / baseline).toBeGreaterThan(RETENTION_FLOOR);
    });
  }
});

The annotation is not decoration. Emit rung from every nightly run and you get a degradation curve over time, which is the artifact that actually changes decisions — a prompt refactor that lifts the 5-document score while collapsing the 40-document score is a regression your averages will hide.

On the insurer’s suite, the first honest run looked like this:

The fix wasn’t a better model. It was ranking the exclusions document into the top of the assembled context instead of letting recency order decide, plus raising the budget after confirming the tokens were affordable. The ladder is what made the tradeoff visible; before it, the team had been arguing about model choice for a month.

Where do you put the needle?

At the depth that hurts. If your ladder passes but production still complains, the position sweep is where you’ll find why.

const DEPTHS = [0, 0.25, 0.5, 0.75, 1];
const scores: number[] = [];

for (const depth of DEPTHS) {
  test(`needle at ${depth * 100}% depth`, async ({ page }) => {
    const claim = await seedClaim({ documents: 40, needle: 'policy-exclusions', depth });
    scores.push(await askAndGrade(page, claim));
  });
}

test.afterAll(() => {
  // Assert the SPREAD, not each score. Position sensitivity is the bug.
  expect(Math.max(...scores) - Math.min(...scores)).toBeLessThan(0.1);
});

Assert the spread, not each score

Asserting the spread rather than each individual score is deliberate. Any one depth failing tells you the model had a bad run; all five depths passing except the middle two tells you your ordering strategy is wrong and gives you a fix — pin the critical document to the head or the tail of the context, where attention actually lands.

Which lane does each test belong in?

Split by determinism, not by importance. The assembly tests never touch a model, so they block merges. The ladder and the sweep score real output, so they alert instead.

That last row is the one to put in front of a product manager. “Our assistant is reliable up to 55 documents; 4% of claims exceed that” is a sentence a business can act on. “The LLM sometimes hallucinates” is not.

If you already instrument spans, the assembly receipt belongs on one — gen_ai.usage.input_tokens plus your dropped-document count in the same span turns every production incident into a query rather than an investigation. That’s the same argument as correlating Playwright tests with GenAI traces, extended one field.

How is this different from a retrieval recall test?

It’s the next failure downstream. Retrieval recall asks whether the right chunk made it into the candidate set. This asks whether the right chunk survived assembly and was still attended to once it was surrounded by 79 other documents. You can have perfect recall@10 and still ship the bug in this post, because recall is measured before truncation and before the model reads anything.

It’s also distinct from multi-turn drift. Instruction adherence decaying by turn 8 is a conversation-length problem; this is a payload-size problem that shows up on turn one. They compound — a long conversation about a large record hits both — but they need different fixtures, and conflating them produces a test that fails for reasons you can’t attribute.

What I’d do differently

I’d have derived the fixture size from production on day one. The four-document fixture wasn’t a decision anyone made; it was the number that fit in the seed file someone wrote in week two, and it silently became the definition of “tested.” One query against the claims table would have shown a p95 of 63 and made the gap obvious before a single line of eval code existed.

I’d also skip the model comparison. We spent close to a month benchmarking alternatives for a failure that turned out to be a break statement in a trim loop. The deterministic assembly test costs an afternoon and rules out half the search space. Run it first, always — the same discipline as putting a token budget on your AI agents before you start tuning prompts for cost.

Your one concrete next step: query your production table for the p95 count of whatever your AI feature loads — documents, messages, line items, attachments — and compare it to the count in your largest test fixture. If the fixture is smaller, you have never tested your feature at the size it actually runs at, and everything above is just describing your next incident.