Your Agent Obeys Turn 1 and Forgets It by Turn 8

A user tells your support agent in turn 2: “Don’t ever charge my card without confirming first.” The agent replies “Understood, I’ll always confirm.” Seven turns later the user asks it to renew a subscription, and it silently runs the charge. Your Playwright suite is green — because it has one test that checks the agent acknowledges the constraint in the turn right after it’s set. It never checks whether the agent still remembers by turn 9. That gap between “honored the rule once” and “honors the rule for the whole session” is where the most expensive agent bugs live, and single-turn tests are structurally blind to it.

The fix is to stop testing turns and start testing sessions. A constraint set early in a conversation is a promise the agent has to keep for the rest of it. The pattern is to pin that constraint in an early turn, drive the conversation forward, and then — much later — fire a probe turn that can only succeed if the constraint is still alive in the model’s working memory. Assert on what the agent does on that probe, not on whether it once said “understood.”

Why do single-turn agent tests pass while production fails?

Because correctness in a conversation is a property of the whole session, not of any one turn — and model reliability collapses as the conversation grows. A May 2025 Microsoft Research and Salesforce study, LLMs Get Lost In Multi-Turn Conversation, simulated over 200,000 conversations and found an average 39% accuracy drop when the same task was delivered across multiple turns instead of one fully-specified prompt. Their decomposition is the important part: aptitude barely moves, but unreliability spikes. “When LLMs take a wrong turn in a conversation, they get lost and do not recover.”

Your single-turn test measures aptitude — can the model do the thing when everything it needs is right there in one message. Production measures reliability across a drifting, growing context. Those are different variables, and the second one is the one your users actually experience.

I watched exactly this play out at a healthcare-scheduling client. A patient could say “I can’t do mornings — I work nights” at the start of a booking flow. The E2E test asserted the agent’s immediate reply (“Got it, afternoons only”) and stayed green for two months. The bug: once the conversation ran long enough — insurance questions, a reschedule, a provider swap — the morning constraint fell out of the model’s effective context and it happily offered an 8 a.m. slot. Every single-turn test passed. The patient-facing behavior was broken the whole time.

Where in the conversation does the constraint actually get lost?

In one of three places, and knowing which one tells you how to force the bug in a test. The constraint doesn’t evaporate randomly; it’s evicted, ignored, or dropped by a mechanism you can reproduce.

The first is context-window eviction. Most chat backends keep a rolling window and drop or summarize the oldest turns once the token budget is hit — the FIFO-queue model that systems like MemGPT formalize as paging between “main context” and “external context.” Your turn-2 constraint is the oldest thing in the buffer, so it’s the first to go. The second is attention degradation: even when the constraint is still in the window, the model may not use it. Stanford’s Lost in the Middle showed models use information best at the very start or very end of the context and worst in the middle — which is exactly where a constraint from turn 2 sits by turn 8. The third is lossy summarization or retrieval, where a memory-compaction step or a RAG layer paraphrases the history and quietly drops the qualifier (“no mornings” becomes “wants afternoons,” which reads the same until the agent offers a morning anyway).

How do you drive a real multi-turn conversation in Playwright?

Build one say() helper that sends a single turn and waits on the assistant’s completion signal — never a timeout — then compose your session out of it. The reason multi-turn tests are rare isn’t that the assertions are hard; it’s that driving a stateful conversation reliably is fiddly, and people give up and write one-shot tests instead.

The trap is the same token-stream race that breaks single-turn AI tests, except now it compounds: if turn 3 asserts before turn 3 finished streaming, every later turn is racing corrupted state. Wait on an explicit done marker your UI already emits.

import { Page, expect } from '@playwright/test';

export async function say(page: Page, text: string) {
  const priorTurns = await page.getByTestId('assistant-turn').count();
  await page.getByRole('textbox', { name: 'Message' }).fill(text);
  await page.getByRole('button', { name: 'Send' }).click();

  const reply = page.getByTestId('assistant-turn').nth(priorTurns);
  await expect(reply).toHaveAttribute('data-complete', 'true'); // completion signal, not sleep()
  return reply;
}

With that one helper, a session becomes readable and every turn is properly synchronized:

export async function chitchat(page: Page, turns: number) {
  for (let i = 0; i < turns; i++) {
    await say(page, `Quick follow-up ${i}: what else should I know?`);
  }
}

How do you assert a constraint set early is honored late?

Use the constraint-probe pattern: set the constraint, advance the conversation, then issue a late turn engineered so the only way to answer it correctly is to still remember the constraint — and assert on the agent’s action, not its acknowledgement. The probe is the whole trick. A good probe is a request the agent will get wrong if, and only if, it has forgotten.

Start with a text probe

Here’s the naive first cut — a text probe. It’s a real test and it catches the loud failures, but treat it as a starting point:

test('honors a dietary constraint set 6 turns earlier', async ({ page }) => {
  await openChat(page);
  await say(page, 'Plan me a week of dinners. I am strictly vegetarian.');
  await chitchat(page, 5); // bury turn 1 under ordinary back-and-forth

  const reply = await say(page, 'Add one high-protein main for Friday.');
  await expect(reply).not.toContainText(/chicken|beef|pork|salmon/i);
});

Upgrade to an assertion on the action

The problem with the text version is the same one I’ve written about in asserting on tool calls instead of replies: the model can say “here’s a great tofu dish” while the structured add_meal call it emits carries chicken. Use Playwright’s request interception to point the assertion at the action:

let addMeal: any = null;
await page.route('**/api/agent/tools/add_meal', async (route) => {
  addMeal = route.request().postDataJSON();
  await route.fulfill({ json: { ok: true } });
});

await say(page, 'Add one high-protein main for Friday.');
await expect.poll(() => addMeal).not.toBeNull();
expect(addMeal.arguments.tags).toContain('vegetarian'); // behavior, not prose

For fuzzier constraints that don’t map to a clean enum — “keep the tone formal,” “don’t mention competitors” — fall back to a semantic check with an LLM-as-judge scoped to the single question “does this response violate the stated constraint, yes or no.” Keep the judge’s job that narrow or you rebuild the flakiness you’re trying to kill.

How do you force the context-window overflow bug on purpose?

Don’t wait for a naturally long chat to trigger eviction — manufacture the pressure. Pad the conversation with enough filler turns to push the early constraint out of the model’s effective window, then probe. This is the test that separates “we handle multi-turn” from “we handle multi-turn under real load,” and it’s the one production keeps finding for you if you don’t write it first.

test('constraint survives context-window pressure', async ({ page }) => {
  await openChat(page);
  await say(page, 'Never contact me by phone — email only.');

  // Bury turn 1 under enough tokens to force eviction/summarization
  for (let i = 0; i < 40; i++) await say(page, longFillerQuestion(i));

  let notify: any = null;
  await page.route('**/api/agent/tools/schedule_notification', async (route) => {
    notify = route.request().postDataJSON();
    await route.fulfill({ json: { ok: true } });
  });

  await say(page, 'Set up my appointment reminders.');
  await expect.poll(() => notify).not.toBeNull();
  expect(notify.arguments.channel).toBe('email'); // not 'sms', not 'call'
});

Tune the filler-turn count to your app’s actual window and compaction threshold — the token budget you track in CI tells you roughly where eviction kicks in. The goal is to land the probe just past the point where turn 1 gets summarized or dropped, because that’s precisely where real conversations cross the line.

Won’t these multi-turn tests just be flaky?

They’ll be variable — but variability is the property under test, so measure it instead of hiding from it. The mistake is running a session probe once and treating one green as a pass. Reliability across repeated attempts is the metric that matters, and it’s brutal: on Sierra’s τ-bench, even an agent with over 60% average task success sees its pass^8 — the odds of getting the same task right on all eight tries — fall below 25%. A single green run tells you almost nothing about the ninth conversation.

Sierra’s own write-up of the benchmark frames pass^k as a deliberate measure of worst-case reliability — the exact thing a single green run papers over. So run the probe k times and gate on a floor, not on a coin flip:

// Reliability IS the unit under test — run the probe k times, gate on a floor.
const K = 8;
const results: boolean[] = [];
for (let i = 0; i < K; i++) results.push(await runConstraintProbe(page));

const passRate = results.filter(Boolean).length / K;
expect(passRate).toBeGreaterThanOrEqual(0.9); // a pass^k floor beats one lucky run

For the PR lane, stub the model responses so the test is deterministic and fast, and reserve the live-model pass^k sweep for a nightly job — the same split I use for output-contract tests on AI features. The stubbed run guards the plumbing on every commit; the nightly sweep guards the behavior that only a real model can break.

What this pattern doesn’t catch

The constraint-probe pattern guards retention — a rule set once and held. It doesn’t test conflict resolution (the user changes the constraint mid-session and the agent has to honor the newer one), and it doesn’t test cross-session memory (does the rule persist after the user closes the tab and comes back). Those are separate probes with separate golden expectations. And none of this replaces a live end-to-end run against the real model in a nightly lane — stubs prove your assertions are wired correctly, but only a real conversation with a real context window proves the agent actually remembers.

Your one next step this week

Take the single most dangerous standing instruction a user can give your agent — the “never charge without confirming,” the “email only,” the hard budget cap — and write exactly one constraint-probe test around it. Set it in turn 1, run ten filler turns, probe on turn 12, assert on the tool call. Run it eight times. If it doesn’t come back 8/8, you’ve just found the bug your single-turn suite has been hiding, and you found it before a customer did.