Gate Time-to-First-Token, Not Total Latency

Your AI chat feature has a latency SLO — p50 total response under two seconds — and it’s green. Meanwhile a user clicks “Send,” watches a blank box for a second and a half, and decides the thing is broken. Both are true at once, and the reason is that you’re measuring the wrong number. On a streaming feature, the latency that decides whether the experience feels alive is time-to-first-token, not total time. Your suite almost certainly gates on the one users don’t feel.

I watched this cost a team a full sprint. A dev added a synchronous auth-token refresh right before the stream opened. Total latency barely moved — the refresh took 300ms and the model still finished in about the same wall-clock time — so the SLO stayed green. But TTFT p95 went from ~700ms to ~2.4s, and the “explain this” panel went from feeling instant to feeling frozen. Support tickets, not a red build, is how they found out. This is the gap between the metric you track and the metric users live in, and it’s exactly the kind of metric that lies while pointing the right direction.

Why does total latency hide a bad streaming experience?

Because streaming decouples the first token from the last one, and users judge responsiveness on the first. A non-streamed request has one moment that matters: it’s done. A streamed one has two — when it starts answering and when it finishes — and human perception gates almost entirely on the first. Nielsen Norman Group’s response-time limits have held for decades: 0.1s feels instant, 1s keeps the user’s flow of thought intact, and past 10s you’ve lost their attention. A stream that paints its first token at 400ms and finishes at 3s sits inside the flow limit the whole way. A stream with the same 3s total but a 1.8s TTFT breaks flow before a single word appears.

This isn’t a rounding error. Perceived performance gates on time-to-first-meaningful-pixel, not time-to-complete — a first token at 180ms reads as faster than a whole answer delivered at 600ms. And the stakes compound outside the lab: Amazon’s classic finding was that every 100ms of added latency cost 1% in sales, and the same abandonment curve applies the second your AI feature makes someone wait on a blank panel.

What exactly should you measure — TTFT, ITL, and total?

Three numbers, and they fail in different ways. Measuring only the last one is how the blank-box regression ships green.

• TTFT (time-to-first-token) — from the send action to the first rendered character. This is the “did it wake up” metric and the one users feel most. Anyscale’s serving metrics guide defines it as request-start to first output token; on the client, measure it to first visible token.
• ITL (inter-token latency) — the gap between consecutive tokens. Low TTFT with high ITL is a stream that starts fast and then stutters — the jank users describe as “it keeps freezing.” Total latency hides this completely.
• Total — send to last token. Still worth tracking, but as a ceiling, not the headline.

For calibration on what “good” looks like in 2026: current inference benchmarks put Groq’s Llama 4 405B at roughly a 0.18s TTFT p50 at 480 tokens/sec — but that’s raw serving latency. By the time you add network egress (US-East to APAC alone adds 180–220ms p50), your gateway, and frontend hydration, the number your user experiences is several times that. And if the feature routes to a reasoning model, TTFT carries a 5–30× tax while the model thinks before it speaks — which is precisely why a chat UX cannot silently fall back to reasoning mode without a gate catching it.

How do you measure time-to-first-token in Playwright?

Two ways, and you want both: the perceived metric users feel, and the transport metric that tells you where a regression came from. Start with perceived, because it’s Playwright-native and it’s the one that maps to the SLO you actually care about.

Perceived TTFT: a stopwatch to the first character

The perceived measurement is just a stopwatch around “click Send” to “first non-empty character in the answer node.” The trick is to wait for the first character, not the finished response — the opposite of what you do when asserting on a completed streaming answer.

test('first token arrives within the flow-limit budget', async ({ page }) => {
  await page.goto('/support');
  await page.getByRole('textbox', { name: 'Ask a question' }).fill('How do I get a refund?');

  const start = Date.now();
  await page.getByRole('button', { name: 'Send' }).click();

  const answer = page.getByTestId('assistant-message');
  // First character, NOT the finished answer — this is TTFT, not total.
  await expect.poll(async () => (await answer.innerText()).trim().length)
    .toBeGreaterThan(0);

  const ttft = Date.now() - start;
  expect(ttft, `TTFT ${ttft}ms exceeded the 1s flow limit`).toBeLessThan(1000);
});

Transport TTFT: timestamp every chunk at the reader

The transport measurement hooks the streaming reader itself, so you get the true first-chunk timestamp and every inter-token gap. Inject it before navigation with addInitScript, wrap fetch, and tee the ReadableStream reader so you timestamp each chunk as it arrives — SSE is the transport under nearly every LLM stream, so each chunk is a real token boundary.

export const installTtftProbe = (page) => page.addInitScript(() => {
  const probe = { ttft: -1, itl: [] as number[] };
  (window as any).__ttft = probe;
  const origFetch = window.fetch;
  window.fetch = async (...args) => {
    if (!String(args[0]).includes('/api/chat')) return origFetch(...args);
    const start = performance.now();
    const res = await origFetch(...args);
    const reader = res.body!.getReader();
    let last = start;
    const stream = new ReadableStream({
      async pull(controller) {
        const { done, value } = await reader.read();
        if (done) return controller.close();
        const now = performance.now();
        if (probe.ttft < 0) probe.ttft = now - start;   // first chunk
        else probe.itl.push(now - last);                // subsequent gaps
        last = now;
        controller.enqueue(value);
      },
    });
    return new Response(stream, { headers: res.headers, status: res.status });
  };
});

Now await page.evaluate(() => window.__ttft) hands you the transport TTFT and the full ITL array. When perceived TTFT regresses but transport TTFT is flat, the model is fine and your frontend got slower — the blocking-refresh bug from the intro shows up exactly here.

Why measure p95 and not the average?

Because the average is the number that lied to the team in the first place — a mean smears the slow requests into the fast ones and reports a lie users never experience. Tail latency is where streaming pain actually lives, and it amplifies with scale. Dean and Barroso’s The Tail at Scale is the canonical read: when a request fans out across backends, per-backend p99 hiccups combine into a much worse end-to-end tail, and they showed hedging at the 95th percentile cutting a BigTable lookup’s p99 from 1800ms to 74ms. Your AI gateway, retrieval layer, and model provider are exactly that kind of fan-out.

So gate on p95 TTFT over a batch of runs, not a single measurement and not the average. One green run tells you almost nothing about a distribution with a fat tail.

const percentile = (xs: number[], p: number) => {
  const s = [...xs].sort((a, b) => a - b);
  return s[Math.ceil((p / 100) * s.length) - 1];
};

test('TTFT p95 stays under budget', async ({ page }) => {
  const samples: number[] = [];
  for (let i = 0; i < 20; i++) samples.push(await measureTtft(page));
  const p95 = percentile(samples, 95);
  expect(p95, `TTFT p95 ${p95}ms over ${samples.length} runs`).toBeLessThan(1200);
});

Where does a TTFT gate belong — the PR pipeline or nightly?

Both, split by what each is allowed to measure. Conventional wisdom — which I mostly agree with — says keep performance tests out of the PR pipeline because they’re slow and flaky under shared CI load. A full load test absolutely stays out. But a single-user TTFT smoke gate is the exception that earns its place inside the pipeline, as long as you’re strict about what it’s measuring.

The split resolves the tension cleanly: the PR gate never touches a real model, so it’s not flaky and it’s not slow, but it still fails the merge that would have regressed the first token. The nightly job owns the noisy, real-world tail.

One real gotcha before you ship this: reasoning models stream a token, go quiet for a beat while they think, then resume. That quiet period is a legitimate high ITL, not a stall — so alert on ITL tails on the nightly job where you know the real model, but keep the deterministic PR gate on TTFT, where the mock removes the ambiguity.

Your one next step

Pick your most-used AI feature and add a single perceived-TTFT probe to its existing spec — the stopwatch around click-to-first-character, nothing more. Run it 20 times locally and look at the spread, not the average. You’ll almost certainly find a p95 that’s 2–3× your p50, and that gap is the experience your total-latency SLO has been hiding. Once you’ve seen your own tail, wire the mocked-stream version into the PR gate in report-only mode and let it watch for a fortnight before you turn on enforcement.