Your Semantic Cache Test Passes. The Cache Still Leaks.

User A asks your assistant “how do I cancel my subscription?” and gets a good answer. Twenty seconds later User B — a different customer, different tenant — asks “how do I cancel my order?” and gets the exact same answer, because a semantic cache decided the two questions were close enough. The end-to-end test is green: the answer rendered, it read plausibly, the assertion matched a heading. Nothing in your suite noticed that B was handed a response that was never generated for B’s question.

A semantic cache quietly rewrites the most important predicate in your app. is this string equal? becomes is this embedding close enough?, and “close enough” is a correctness boundary — sometimes a security boundary — that your DOM assertions structurally cannot see. Teams bolt these caches on to cut token spend, then ship the cache with tests that only prove the happy path renders. This post is about the two failure modes those tests miss and the Playwright patterns that catch them.

What is a semantic cache false hit?

A false hit is when the cache returns a stored answer for a query that isn’t actually equivalent to the one that produced it. Instead of hashing the prompt for an exact match, a semantic cache embeds the query, runs a similarity search, and serves the cached response when the score clears a threshold. Everything above the line is treated as “the same question.” The whole trick — and the whole risk — lives in where you draw that line.

Tools like GPTCache and Redis LangCache exist because exact-match caching barely hits — real users phrase the same intent a hundred ways. But loosening the match to catch paraphrases also catches near-misses that share vocabulary and diverge in meaning. “Reset my password” and “reset my API key” sit a hair apart in embedding space. So do “refund policy for EU customers” and “refund policy for US customers” — one word different, materially different answer, and a similarity score high enough to collide.

The numbers are unforgiving. At a loose 0.70 threshold, measured cache precision drops to roughly 0.9 — about one in ten hits answers a different question than the user asked. Production guidance from the Redis LangCache team is blunter: once your false-positive rate crosses 3–5%, threshold tuning alone won’t save you and you need an architectural fix. A 3% wrong-answer rate would be a P1 in any other part of your stack. Bolted in front of an LLM, it ships as a performance optimization.

Why doesn’t your end-to-end test catch it?

Because your assertion checks that an answer appeared, not that it was this user’s answer. The DOM is the weakest oracle in an AI feature — I made that case at length in asserting on the GenAI trace instead of the DOM, and a cache false hit is the cleanest example of why. A false hit produces a well-formed, fluent, on-topic-looking response. It fills the div. getByTestId('answer') finds text. toBeVisible() passes. Every signal your test reads is the same whether the answer was generated for this query or lifted from a neighbor.

The provenance — was this freshly generated, was it scoped to this tenant, does it belong to this question — never crosses into the browser. It lives in a cache-hit flag and a scoping key on the server. Which is exactly how these bugs survive code review.

How do you test for a false cache hit in Playwright?

Stop asserting that an answer exists and start asserting on its provenance — cache status and identity. You need two things the DOM won’t give you: a machine-readable hit/miss signal, and a corpus of near-miss pairs that must never collide.

Assert on provenance, not presence

The test itself is short because the corpus carries the intent:

import { test, expect } from '@playwright/test';
import { ask, cacheStatus } from './helpers/assistant';

// Pairs that share words but mean different things — neither may reuse the other.
const nearMisses = [
  ['How do I cancel my subscription?', 'How do I cancel my order?'],
  ['Reset my password',               'Reset my API key'],
  ['Refund policy for EU customers',  'Refund policy for US customers'],
];

for (const [seed, probe] of nearMisses) {
  test(`"${probe}" must not reuse the answer to "${seed}"`, async ({ page }) => {
    const seedRes  = await ask(page, seed);   // warm the cache
    const probeRes = await ask(page, probe);  // the near-miss query

    // The DOM shows a plausible answer either way — assert on provenance.
    expect(await cacheStatus(probeRes)).toBe('MISS');
    expect(probeRes.answer).not.toBe(seedRes.answer); // byte-identical = false hit
  });
}

The cacheStatus assertion is the robust one. The byte-identity check is a cheap backstop: two genuinely distinct generations should never come back character-for-character equal, so equality is a reliable smell for “the cache handed B what it made for A.” If your model runs at a non-zero temperature this backstop is even stronger — and if it doesn’t, remember that temperature zero is still not deterministic, so never lean on output equality as a positive correctness check, only as a false-hit alarm.

How do you prove the cache is tenant-isolated?

Drive two isolated browser contexts as two different tenants, send the same query from both, and assert the second tenant gets a MISS. This is the test the vLLM team was missing. Same-string queries from different tenants are the maximum-similarity case — if soft embedding-distance scoping is going to leak, it leaks here, every time.

Two Playwright contexts give you two fully separated cookie jars and auth sessions in one worker, which is exactly the isolation primitive you want — the same reason contexts beat a shared browser for parallel state isolation. Seed as tenant acme, probe as tenant globex:

import { test, expect } from '@playwright/test';
import { asTenant, ask, cacheStatus } from './helpers/assistant';

test('an identical query from another tenant gets a cache MISS', async ({ browser }) => {
  const acme   = await asTenant(browser, 'acme');    // isolated context + auth
  const globex = await asTenant(browser, 'globex');

  const seed  = await ask(acme,   'Summarize my last invoice');
  const probe = await ask(globex, 'Summarize my last invoice'); // same string, other tenant

  // Soft (embedding-distance) scoping will happily serve acme's answer to globex.
  expect(await cacheStatus(probe)).toBe('MISS'); // fresh generation, per tenant
  expect(probe.answer).not.toBe(seed.answer);    // identical answer here = CWE-524
});

This same trap shows up without a cache the moment tenants share fixtures — I dug into that failure mode in shared test users breaking parallel Playwright suites. The cache just makes it invisible: the leak is now an optimization firing correctly, not a state bug. The fix is not a tighter threshold. It’s a hard partition key.

- const key = embed(query);                       // similarity search across everyone
+ const key = `${tenantId}:${locale}:${model}`;   // hard partition prefix
+ // run the similarity search *within* this partition only, never across it

Should these tests block a merge or run nightly?

Split them by determinism. The tenant-isolation test is fully deterministic — same tenant, same query, one correct outcome — and it guards a security boundary, so it belongs in the blocking gate. The near-miss precision corpus depends on the embedding model and threshold, so it’s better run as a budgeted eval that fails on a rate, the second lane in a mock-in-CI, eval-nightly split.

Thresholds are workload-specific

That division matters because thresholds are workload-specific, not universal. Published starting points range from 0.94 for FAQ lookups up to 0.97 for transactional queries where the tolerance for a wrong answer is basically zero. A single global threshold across a product with both is a guarantee that one surface is either leaking or never hitting. Your eval lane is where you discover which.

What I’d do differently

The first time I owned one of these, we treated the false-hit rate as a threshold-tuning exercise. We nudged the similarity cutoff up until the wrong answers stopped — and watched the hit rate collapse and the token bill climb right back to where it was before we added the cache. We’d optimized the metric and deleted the value. The actual fix was structural: hard partition keys on tenant, locale, and model, with similarity matching confined inside each partition. False hits and cross-tenant leaks both went to zero and the hit rate held, because we stopped asking one number to enforce a boundary it was never shaped to enforce. This mirrors how GPTCache’s own weakness is measured against stricter designs — precision problems are architectural, not a slider you can drag to safety.

I’d also have built the near-miss corpus from real traffic on day one instead of inventing clever pairs at my desk. The collisions that hurt are the ones your users actually type — questions that live one token apart in your product’s specific embedding space — and you can only find those by mining production queries that scored just under the line. Invented pairs test your imagination; harvested pairs test your cache.

Your one concrete next step: write the tenant-isolation test above — two contexts, identical query, different tenant IDs — and run it against your staging cache today. If it passes, you’ve proven isolation for one case. If it fails, you just found a CWE-524 before your customer’s security team did, and you have the exact reproduction in hand.