Your AI Tests Pass. Mutation Testing Says They're Fake

You asked an LLM to write tests for a payments module. It produced forty specs, every one green, and your coverage badge jumped from 71% to 94% in an afternoon. Then you ran mutation testing against the same code and the real number came back: the suite catches about a third of the bugs it should. The coverage was real. The protection was theater. That gap — high coverage, low fault detection — is the single most dangerous artifact of AI-generated tests, and mutation testing is the only cheap, automatable way to see it.

This is the post-2025 version of a problem I’ve been fighting for a decade. The patterns that turn a green test suite into a liar used to take a tired engineer a sprint to introduce by hand. An LLM can now generate a thousand lines of them before lunch, and they look more convincing than the human-written kind because the assertions are syntactically perfect. The stakes scale with how much of your suite you’ve handed to a model: the more tests AI writes, the more your green dashboard is measuring the wrong thing.

Why do AI-generated tests pass while still missing bugs?

Because an LLM writes tests that assert what your code does, not what it should do. When it reads an implementation to generate a test, it treats the current behavior as ground truth. If the implementation has a bug, the AI faithfully encodes that bug as the expected result — the test goes green and locks the defect in place. This is the inverse of test-driven development, and it’s the core reason coverage and correctness diverge under AI generation.

The numbers are blunt. One 2025 analysis found that LLM-generated tests scored just 20.32% on mutation testing for complex real-world functions — meaning roughly four out of five injectable bugs sailed through untouched. A developer who had 93.1% reported coverage measured the real figure with mutation testing and got 34%. And the surface area is growing: AI-generated code ships 1.7x to 2.7x more defects than human-written code, so the very tests you’re trusting are guarding the code most likely to be broken.

The two failure modes coverage can’t see

The two specific failure modes I see most in review: tautological assertions (mock a dependency to return { total: 100 }, then assert the function returns 100 — you’ve tested the mock, not the math) and over-mocking, where the AI stubs out every collaborator so nothing real is ever exercised. Both produce green tests and zero protection. Coverage can’t distinguish them from real tests because coverage only measures whether a line ran, never whether anyone checked the result.

What is mutation testing, and why is it the right lie detector?

Mutation testing deliberately injects small bugs — “mutants” — into your source code, then reruns your tests. If a test fails, it “killed” the mutant: good, it would have caught that bug. If every test stays green, the mutant “survived,” and you’ve found an assertion gap. Your mutation score is the percentage of mutants killed, and unlike coverage, it measures detection rather than execution.

The standard mutation operators are exactly the bugs humans and LLMs actually ship. A tool flips > to >=, swaps && for ||, changes + to -, removes a function call, or replaces a return value with a default.

// Original — correct business rule
function applyDiscount(price, discount) {
  if (discount > 0.5) throw new Error('Discount cannot exceed 50%');
  return price * (1 - discount);
}

// Mutant Stryker injects: > becomes >=
// if (discount >= 0.5) throw new Error('Discount cannot exceed 50%');

That boundary mutant is the whole point. A test that only ever passes discount = 0.1 will never notice the >= flip, so the mutant survives — and that survival is your proof that the AI never wrote a boundary case. Coverage would have happily reported the line as covered.

Why killing mutants predicts catching real bugs

This isn’t a fringe heuristic. The foundational study by René Just and colleagues at FSE 2014, Are Mutants a Valid Substitute for Real Faults?, found a statistically significant correlation between a suite’s mutant-detection rate and its real-fault-detection rate — independent of code coverage. In plain terms: killing mutants predicts catching real bugs in a way that coverage simply does not. For JavaScript and TypeScript suites, StrykerJS is the mature tool; treat a mutation score above 80% as a strong suite and anything under 60% as a suite you can’t trust to guard a refactor.

Can you mutation-test a Playwright suite directly?

No — and this is the trap most people hit on day one. Mutation testing reruns your tests once per mutant, and a real codebase generates thousands of mutants. If each run is a 10-minute Playwright E2E pass, you’re looking at hours or days per session. Mutation testing only works when the tests are fast enough to run thousands of times, which E2E tests categorically are not.

So you point it at the parts of your stack that are fast and deterministic — and those are exactly the parts AI loves to generate weak tests for:

The move is to mutation-test the unit and integration layer that covers the same code paths your Playwright suite walks through, and keep Playwright for what it’s uniquely good at: validating real user flows in a real browser. You get fast bug-detection feedback on logic and genuine end-to-end confidence on journeys, without waiting three days for a mutation run.

Performance is no longer the blocker it was. StrykerJS’s coverageAnalysis: "perTest" records which test covers which mutant and reruns only those tests for that mutant, plus incremental mode so CI only re-mutates changed code.

{
  "testRunner": "vitest",
  "coverageAnalysis": "perTest",
  "incremental": true,
  "mutate": ["src/pricing/**/*.ts", "tests/support/**/*.ts"],
  "thresholds": { "high": 80, "low": 60, "break": 60 }
}

The break: 60 line is the one that matters for AI-generated tests — it fails CI when the mutation score drops below 60%, which is your gate against a model quietly adding green-but-empty tests.

How do you wire mutation testing as a gate on AI-generated tests?

You make it a required check on the diff, scoped to changed files, with a threshold that blocks the merge. The discipline is the same one I argued for with letting AI triage failures but never decide them: the model can propose tests all day, but a deterministic gate decides whether they’re real. Mutation score is that gate.

That last step is the inversion that makes AI test generation actually safe. Coverage as a target produces tests that run code; mutation score as a target produces tests that check code. Point the model at survivors and you turn a coverage-inflation machine into a bug-detection machine.

On an anonymized fintech suite where a team had let Copilot generate roughly 300 service-layer tests, coverage read 91% and everyone felt safe. The first scoped Stryker run came back at a 38% mutation score — and within that, the refund-calculation module, the highest-risk code in the app, sat at 19%. Industry tooling reports that mutation testing catches around 60% more subtle faults than coverage-based metrics alone, and this was exactly that gap made visible. Three weeks of killing survivors took the refund module to 84%, and two of the bugs surfaced along the way were already latent in production.

What I’d do differently

The first time I introduced mutation testing on an AI-heavy suite, I tried to run it across the entire repo in one nightly job and report a single number. It took four hours, nobody looked at the report, and the global score averaged away the modules that actually needed help. It was coverage theater with extra steps — a vanity metric that happened to be harder to compute.

What worked was the opposite: small, scoped, blocking, and on the diff. Make mutation score a per-PR gate on changed files, keep it under five minutes with incremental mode, and never let it become a dashboard people glance at and ignore. The number only changes behavior when it can block a merge. And resist the urge to mutation-test your Playwright specs directly — that lesson cost me a wasted weekend. Mutate the logic underneath them; let the browser tests do browser work.

Your one concrete next step

This week, pick the one module your AI tools have written the most tests for — the one with the coverage badge you’re proudest of — and run StrykerJS against just that file with npx stryker run --mutate "src/that-module.ts". Don’t gate anything yet. Just look at the mutation score next to the coverage number. If the gap is as wide as it usually is, you’ll have the only evidence you need to make mutation score a required check. The rest is configuration. For the bigger picture on where AI helps and where it quietly hurts, my write-up on letting AI write a full test suite covers the review workflow that pairs with this gate.