Nothing Validates WebMCP Inputs: Write Tool Contract Tests

Five bookings landed in my demo’s database from tool calls that broke the tool’s own inputSchema. One was for zero people. One was dated 42. Chromium ran every call and handed the input straight to the page’s execute function, and Playwright returned the results as successes. Nothing between the caller and your code checks the schema you published. Only WebMCP tool contract tests will tell you.

A WebMCP tool is a public API that agents call with whatever arguments the model produces, so test it the way you’d test any API: pin the published contract, throw schema-violating input at it, check the error shape, and verify the annotations tell the truth. Below are those four checks in Playwright TypeScript and Playwright Java, reading one shared contract file. Every result comes from a demo I ran on Playwright 1.65.0 for TypeScript (Chromium 158) and Playwright Java 1.64.0 (Chromium 156), the latest Java release on Maven Central.

Why do WebMCP tools need contract tests now?

Because both Playwright languages can now call them directly, and agents already do. Playwright 1.64 shipped page.webmcp on October 7, and Playwright Java 1.64.0 followed on October 21 with page.webmcp(). Playwright MCP exposes the same page tools to agents as webmcp_<tool> by default.

A page registers a tool through WebMCP with document.modelContext.registerTool(): a name, a description, a JSON Schema for input, optional annotations, and an execute callback. To an agent, that schema is the interface definition. To your backend, the tool is one more client that can send it anything.

Most WebMCP testing write-ups stop at listing the tools and calling one with valid input. That’s a smoke test. In the post on agents routing around a broken form, I argued tool tests belong in their own deterministic lane. This post builds that lane.

Does the browser validate WebMCP tool input against inputSchema?

No. Neither the browser nor Playwright checks callTool() arguments against the schema the page registered. The WebMCP spec’s executeTool algorithm serializes the input object to JSON and dispatches it. It checks origin, registration and the target document, but nothing about the schema. Validation is the tool author’s job, and nothing tells you that until you test it.

The demo: a meeting-room booking page

The fixture is a small page I’ll call Roomly, served from localhost (a secure context, which WebMCP requires). It registers two tools: search_rooms (annotated read-only) and book_room (annotated consequential). book_room registers after the room list loads, the way most apps hydrate:

const bookSchema = {
  type: 'object',
  properties: {
    roomId: { type: 'string', enum: rooms.map((room) => room.id) },
    date: { type: 'string', format: 'date' },
    attendees: { type: 'integer', minimum: 1 },
  },
  required: ['roomId', 'date', 'attendees'],
};
document.modelContext?.registerTool({
  name: 'book_room',
  description: 'Book a meeting room for one day.',
  inputSchema: bookSchema,
  annotations: { consequentialHint: true },
  execute: async (input) => { /* POST /api/bookings, return { content } or { content, isError: true } */ },
});

The execute body is the usual front-end code: it posts to the backend and maps a non-2xx response to isError: true. It does no validation of its own.

What eleven schema-violating calls did

I generated one invalid input per schema rule (more on that below) and sent all of them through page.webmcp.callTool():

Three rejections out of eleven, and all three came from the backend rejecting an unknown room, not from anything that read the schema. If your API behind the tool is strict, you’re partly covered by accident. If it trusts the front end, the tool is an open door.

What should a WebMCP tool contract test cover?

Four things, the same four I’d want from any API that third parties call:

The golden file is the piece I’d fight for in review. Put it in a plain JSON file, contract/roomly-tools.json, not in a framework-specific snapshot directory. Then the TypeScript suite and the Java suite assert against the same bytes, and a contract change shows up as one reviewable diff.

How do you set up page.webmcp in Playwright TypeScript and Java?

Launch Chromium with --enable-features=WebMCP, navigate, and poll until the tool count matches the contract before you assert anything. Without the flag, tools() throws. Without the poll, you’re racing the page’s registration.

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  workers: 1,
  use: {
    baseURL: process.env.BASE_URL ?? 'http://localhost:4420',
    launchOptions: { args: ['--enable-features=WebMCP'] },
  },
});

The fixture loads the golden contract once and hands every test a fully registered tool list:

import { readFileSync } from 'node:fs';
import { test as base, expect, type Page } from '@playwright/test';

type Tool = Awaited<ReturnType<Page['webmcp']['tools']>>[number];
export const contract: Tool[] = JSON.parse(readFileSync('../contract/roomly-tools.json', 'utf8'));

export const test = base.extend<{ tools: Tool[] }>({
  tools: async ({ page }, use) => {
    await page.goto(process.env.APP_PATH ?? '/');
    // Tools register as the app hydrates, so the first tools() call can come back short.
    await expect.poll(async () => (await page.webmcp.tools()).length).toBe(contract.length);
    await use(await page.webmcp.tools());
  },
});
export { expect };

Java has no expect.poll, but page.waitForCondition() does the same job. JUnit 5, Playwright Java 1.64.0:

@BeforeEach
void openApp() {
  api = playwright.request().newContext(new APIRequest.NewContextOptions().setBaseURL(BASE_URL));
  api.post("/api/reset");
  page = browser.newPage();
  page.navigate(BASE_URL + APP_PATH);
  // Tools register as the app hydrates; poll instead of trusting the first tools() call.
  page.waitForCondition(() -> page.webmcp().tools().size() == 2);
  tools = page.webmcp().tools();
}

The browser is launched once in @BeforeAll with new BrowserType.LaunchOptions().setArgs(List.of("--enable-features=WebMCP")).

The registration race is real

My very first probe called tools() straight after goto() and got one tool back, search_rooms, because book_room hadn’t registered yet. Re-running that probe in two batches of 20 loads, tools() came back short 2 times and then once: 3 of 40. That’s the worst kind of race: it passes most runs on your machine and flakes on a loaded CI runner. Poll for the count you expect, and if you’ve read why tests pass on retry, you already know why.

How do you catch WebMCP schema drift between releases?

Compare the published tool list to the golden file with deep equality. It catches renamed fields, dropped required entries, changed descriptions and lost annotations in one assertion, with a diff that tells the reviewer exactly what moved.

import { test, expect, contract } from './webmcp-fixture';

test('published tool contract has not drifted', async ({ tools }) => {
  const published = tools.map(({ name, description, inputSchema, annotations }) =>
    ({ name, description, inputSchema, annotations }));
  expect(published).toEqual(contract);
});

To simulate next sprint’s “harmless refactor”, the demo page has a variant that renames attendees to headcount and drops the read-only annotation from search_rooms. The contract test fails with this (trimmed) diff:

         "attendees": Object {
           "minimum": 1,
           "type": "integer",
         },
         "headcount": Object {
           "minimum": 1,
           "type": "integer",
...
         "attendees",
         "headcount",
...
     "annotations": Object {
       "readOnly": true,
     },
     "annotations": undefined,

Every agent prompt, eval and Playwright MCP session that learned attendees breaks on that rename. It’s the same class of break as the userName to fullName rename in contract testing vs API testing, except your consumer is a model that won’t open a ticket. Note the last hunk too. When the page passes annotations: {}, Playwright reports annotations as undefined, not an empty object.

The Java version reads the same file

@Test
void publishedToolContractHasNotDrifted() throws Exception {
  JsonElement expected = JsonParser.parseString(Files.readString(Path.of("../contract/roomly-tools.json")));
  JsonElement actual = new Gson().toJsonTree(tools);
  assertEquals(expected, actual, () -> "WebMCP contract drifted:\n" + actual);
}

Gson is already on the classpath as a Playwright Java dependency. Compare parsed JsonElements, not strings. Playwright Java hands back inputSchema as a plain Object of maps whose numbers are doubles, so Gson serializes "minimum":1 as "minimum":1.0. String comparison against the golden file fails on every integer. JsonElement.equals compares numbers by value and passes. The WebMCPToolAnnotations fields are boxed Booleans, null when unset, so Gson omits them and the output matches the TypeScript shape exactly.

How do you generate negative tests from a WebMCP inputSchema?

Read the schema the tool publishes and derive one violating input per rule: drop each required field, send the wrong type for each property, step outside each enum, and go one below each minimum. Then assert every one comes back as isError with no side effect. The schema is the spec, so the test stays in sync with it for free.

type Schema = { properties?: Record<string, any>; required?: string[] };

// Derive inputs that violate the tool's own inputSchema, one rule at a time.
export function invalidInputs(schema: Schema, valid: Record<string, unknown>) {
  const cases: { why: string; input: Record<string, unknown> }[] = [];
  for (const field of schema.required ?? []) {
    const { [field]: _dropped, ...rest } = valid;
    cases.push({ why: `missing required "${field}"`, input: rest });
  }
  for (const [field, rule] of Object.entries(schema.properties ?? {})) {
    const wrongType = rule.type === 'string' ? 42 : 'not-a-number';
    cases.push({ why: `"${field}" is not ${rule.type}`, input: { ...valid, [field]: wrongType } });
    if (rule.enum) cases.push({ why: `"${field}" outside enum`, input: { ...valid, [field]: 'R-999' } });
    if (rule.minimum !== undefined)
      cases.push({ why: `"${field}" below minimum`, input: { ...valid, [field]: rule.minimum - 1 } });
  }
  return cases;
}

This handles the keywords my schemas use. Extend it as yours grow (maxLength, pattern, nested objects), or swap in a schema-based fuzzer once you outgrow twenty lines. The test that drives it:

const validInput: Record<string, Record<string, unknown>> = {
  search_rooms: { minSeats: 6 },
  book_room: { roomId: 'R-204', date: '2027-01-12', attendees: 6 },
};

test.beforeEach(async ({ request }) => { await request.post('/api/reset'); });

test('every schema violation is rejected with no side effect', async ({ page, tools, request }) => {
  for (const tool of tools) {
    for (const { why, input } of invalidInputs(tool.inputSchema, validInput[tool.name])) {
      const result = await page.webmcp.callTool(tool.name, input);
      expect.soft(result.isError, `${tool.name}: ${why}`).toBe(true);
    }
  }
  expect(await (await request.get('/api/bookings')).json()).toEqual([]);
});

expect.soft matters here. You want all eleven verdicts in one run, not the first failure. Against the unvalidated page, the report lists eight soft failures by name (book_room: missing required "date", search_rooms: "minSeats" below minimum, and so on), then the hard failure: five bookings in a database that should be empty.

The same generator in Java

The Java helper mirrors it, returning a list of record Case(String why, Map<String, Object> input). The interesting part is the test, which collects accepted violations into one list so the failure reads like a punch list:

@Test
@SuppressWarnings("unchecked")
void everySchemaViolationIsRejectedWithNoSideEffect() {
  var accepted = new ArrayList<String>();
  for (WebMCPTool tool : tools) {
    for (var negative : SchemaNegatives.from(tool.inputSchema, VALID_INPUT.get(tool.name))) {
      var result = (Map<String, Object>) page.webmcp().callTool(tool.name, negative.input());
      if (!Boolean.TRUE.equals(result.get("isError"))) accepted.add(tool.name + ": " + negative.why());
    }
  }
  assertAll(
      () -> assertEquals(List.of(), accepted, "schema violations the tool accepted"),
      () -> assertEquals(0, bookings().size(), "bookings written by invalid input"));
}

callTool() returns Object in Java. For a tool that returns the usual { content, isError } shape, that’s a LinkedHashMap, so the cast is safe. On the unvalidated page, Maven reports the same eight accepted violations and bookings written by invalid input ==> expected: <0> but was: <5>.

The fix belongs in execute, against the same schema object

Register the schema as a constant and validate against that same object inside execute, so the published contract and the enforced one can’t drift apart. In production that’s Ajv compiling the schema once. The demo uses a dependency-free check for its four keywords, wrapped around each tool:

function validated(schema, execute) {
  return async (input) => {
    const errors = violations(schema, input);
    return errors.length ? { ...text({ error: 'INVALID_INPUT', details: errors }), isError: true } : execute(input);
  };
}

With validation on, the TypeScript suite goes from 5 of 6 passing to 6 of 6, and the Java suite from 4 of 5 to 5 of 5. Same tests, no edits.

Should a WebMCP tool throw or return isError?

Return isError: true for every expected failure. Playwright’s docs say a result with isError: true comes back as is, while callTool() throws when the tool’s execute throws. What the docs don’t mention is how little of the thrown error survives:

import { test, expect } from './webmcp-fixture';

test('a tool that throws loses its error message', async ({ page, tools }) => {
  await page.evaluate(() => (document as any).modelContext.registerTool({
    name: 'flaky_lookup',
    description: 'Throws instead of returning isError.',
    execute: async () => { throw new Error('calendar service timed out'); },
  }));
  await expect(page.webmcp.callTool('flaky_lookup')).rejects.toThrow(
    'webMCP.callTool: OperationError: Tool execution failed.');
});

That test passes. “Calendar service timed out” never reaches the caller. An agent gets a generic failure it can’t reason about, and your test report gets the same. Pin the error shape the way you’d pin an HTTP error body:

test('business errors come back as isError, not as a throw', async ({ page, tools }) => {
  const result = await page.webmcp.callTool('book_room', { ...validInput.book_room, roomId: 'R-101', attendees: 9 });
  expect(result).toMatchObject({ isError: true, content: [{ type: 'text', text: '{"error":"OVER_CAPACITY"}' }] });
});

Calling a tool name that isn’t registered is the one case that should throw, and it does in both languages: No WebMCP tool named "cancel_booking". Available tools: book_room, search_rooms. In Java it arrives as a PlaywrightException, so assertThrows(PlaywrightException.class, ...) plus a contains check on that message covers it. If you’ve built fault injection for agent tools, this is the page-side half of the same contract.

How do you test that readOnly annotations tell the truth?

Call every tool the page marks readOnly with valid input, then assert backend state didn’t change. Agents and clients use the read-only hint to decide what’s safe to call without asking. The spec itself warns there’s no guarantee a tool’s declared intent matches what it does.

test('tools annotated readOnly do not write', async ({ page, tools, request }) => {
  for (const tool of tools.filter((t) => t.annotations?.readOnly)) {
    await page.webmcp.callTool(tool.name, validInput[tool.name]);
  }
  expect(await (await request.get('/api/bookings')).json()).toEqual([]);
});

To check it catches something, I gave search_rooms a “helpful” feature in a demo variant: hold the best-matching room for the agent. The test failed with one booking dated "hold" that a read-only search had created.

A filtered test passes when there’s nothing to filter

Run that same test against the drifted release, where search_rooms lost its readOnlyHint, and it passes. The filter finds zero read-only tools, the loop never runs, and an empty bookings table proves nothing. This is why the four checks travel together. The honesty test checks behavior, and the golden contract is what notices the annotation disappeared. Neither one covers both.

How do the TypeScript and Java WebMCP APIs differ?

Same capability, a few differences that matter when you port a suite:

The timeout difference comes straight from the TypeScript and Java API docs, and it’ll bite you. A TypeScript tool call that hangs waits for the test timeout. The same call in Java fails at 30 seconds. Set it explicitly in TypeScript with callTool(name, input, { timeout: 10_000 }). Also note page.webmcp covers the main frame only. Tools registered inside an iframe need that frame’s own webmcp.

If your Java suite already has an API client and data factory layer, the api request context above is where it plugs in. Resetting and reading state through the API is what makes the side-effect assertions trustworthy.

Common errors

• tools() throws on first call. The browser was launched without WebMCP. Chromium needs --enable-features=WebMCP, Firefox needs the dom.modelcontext.enabled and dom.modelcontext.testing.enabled prefs, and WebKit doesn’t implement it.
• Tools never register. The page called navigator.modelContext. The spec and Chrome use document.modelContext, though Playwright’s own class description still says navigator.
• page.evaluate: InvalidStateError: Duplicate tool name. Something registered the same name twice, usually a re-render that re-ran registration without unregistering through the AbortSignal option.

Test your consequential tool first

Pick the one consequential tool your page registers, the one that books, buys, sends or deletes. Write the schema-negatives test for it in whichever language your suite already uses, and run it before you write a single happy-path assertion. If it comes back green, you’ve confirmed a guarantee. If it doesn’t, you’ve found out which layer was really validating input, before an agent finds out for you.

FAQ