AI-Generated Code Ships 1.7x More Bugs Your Tests Miss

AI-assisted pull requests carry 10.83 issues each — 1.7x the human rate — according to CodeRabbit’s December 2025 study of 470 open-source PRs. Logic and correctness errors, the kind that silently corrupt an order total, run 1.75x. And yet the test automation suites guarding that code stay green, because most teams answering “how to test AI-generated code” are letting the same model write the tests.

Two failure distributions are lining up in 2026 test automation, and the overlap is the problem. AI-generated code bugs concentrate in logic, error handling, and I/O behavior. AI-generated tests concentrate their assertions on the happy path — the one place those defect categories don’t appear. DORA’s 2025 report still finds AI adoption negatively associated with delivery stability, and GitClear’s 2026 analysis measured error-masking constructs up 47%. The gap between what AI code breaks and what AI tests check is where your next production incident lives. The fix isn’t more tests. It’s a different assertion strategy — three patterns, covered below.

Why Does AI-Generated Code Have More Bugs Than Human Code?

CodeRabbit compared 320 AI-co-authored PRs against 150 human-only PRs and found 10.83 issues per AI-assisted PR versus 6.45 — roughly 1.7x overall. The distribution matters more than the multiplier: logic and correctness errors were 1.75x more common — the number The Register’s coverage led with — and CodeRabbit’s own category breakdown puts error-handling gaps at roughly 2x and excessive I/O near 8x.

Before you put that number in a slide deck: it’s a vendor study, measured with CodeRabbit’s own review tool, and authorship was inferred from co-author signals; PRs without those signals were assumed human. The human baseline is 150 PRs. I believe the direction, not the third decimal. It also matches what other groups measured independently — Veracode found 45% of AI-generated code introduces security flaws, and METR’s randomized trial found experienced devs were 19% slower with AI while believing they were 20% faster.

That last stat is the whole story in miniature. The failure mode of AI-assisted engineering is confident wrongness that survives review: code that reads clean, compiles, and passes the tests that shipped alongside it.

Why Do AI-Generated Tests Only Cover the Happy Path?

AI-generated tests derive their assertions from the implementation, so they encode what the code currently does as what it should do. A model handed buggy code will generate tests that lock in the bug as expected behavior. That’s not a tooling flaw you can prompt away — it’s the test-oracle problem, and generated tests sit on the wrong side of it.

I measured this on my own suite. When I let AI write my Playwright test suite, 10 of the 14 generated tests covered happy paths or simple validation, and all 4 attempted edge cases were wrong. The “session expiry” test asserted the session token existed — it never expired one. The OAuth redirect test asserted only that we’d left the login page, not where we went. Rereading that suite with CodeRabbit’s taxonomy in hand, the mapping is exact: every broken test sat in one of the report’s top defect categories — logic, error handling, or unverified side effects. The suite wasn’t weak at random; it was weak in precisely the places code generation is most likely to be wrong.

The research says my 14 tests weren’t an outlier. TOGLL, the first large-scale study of LLM-generated test oracles, found roughly a quarter of generated assertion oracles fail against known-correct code — the oracle itself is wrong out of the box. Teams respond to noisy generated assertions the way teams always respond to noise: soften or delete them until CI goes green. What survives that pruning is a green suite hiding real bugs, built at machine speed.

How Should You Test AI-Generated Code in 2026?

Write the assertions yourself, and aim them at the three defect categories where AI code actually fails: logic errors, missing failure handling, and unintended side effects. The test scaffolding — locators, page objects, file structure — is the part AI generates well, so keep delegating that. The oracle is the part it reliably gets wrong.

Each pattern below targets a measured defect category from the CodeRabbit taxonomy.

Pattern 1: Assert invariants, not outputs (logic errors, 1.75x)

Here’s the shape of assertion an AI generates when you ask it to test a cart:

import { test, expect } from '@playwright/test';

// AI-generated: the expected value came from the implementation
test('applies TENOFF coupon', async ({ page }) => {
  await page.goto('/cart');
  await page.getByRole('textbox', { name: 'Promo code' }).fill('TENOFF');
  await page.getByRole('button', { name: 'Apply' }).click();

  await expect(page.getByTestId('cart-total')).toHaveText('$103.00');
});

Looks fine. But where did $103.00 come from? The implementation, which applied the $10 coupon after 13% tax instead of before it. The spec says $101.70. The model read the code, took its output as truth, and enshrined the bug. Green test, wrong math, and no reviewer will catch it because the assertion looks maximally specific.

An invariant assertion comes from the domain, not the code, so it can’t inherit the code’s bugs:

import { test, expect, type Locator } from '@playwright/test';

const money = async (loc: Locator) =>
  parseFloat((await loc.innerText()).replace(/[^0-9.]/g, ''));

test('tax is charged on the discounted subtotal', async ({ page }) => {
  await page.goto('/cart');
  await page.getByRole('textbox', { name: 'Promo code' }).fill('TENOFF');
  await page.getByRole('button', { name: 'Apply' }).click();
  await expect(page.getByTestId('discount')).toBeVisible();

  const subtotal = await money(page.getByTestId('subtotal'));
  const discount = await money(page.getByTestId('discount'));
  expect(discount).toBeLessThanOrEqual(subtotal);
  expect(await money(page.getByTestId('cart-total')))
    .toBeCloseTo((subtotal - discount) * 1.13, 2);
});

This version fails against the buggy implementation no matter what literal the model would have picked, because the relationship between the numbers is the assertion.

Pattern 2: Assert the failure path (error-handling gaps, ~2x)

Error-handling gaps are twice as common in AI code, and generated tests almost never exercise a failure, because there’s no failure in the implementation to derive one from. Force the failure and assert two things: the user sees the truth, and the failure doesn’t impersonate success.

test('checkout refuses to order when payments are down', async ({ page }) => {
  await page.route('**/api/payments/**', route =>
    route.fulfill({ status: 503, body: 'Service Unavailable' }));

  await page.goto('/checkout');
  await page.getByRole('button', { name: 'Place order' }).click();

  await expect(page.getByRole('alert')).toContainText('could not be processed');
  // The assertion AI never writes: failure must not look like success
  await expect(page.getByTestId('order-confirmation')).toHaveCount(0);
});

The second assertion is the load-bearing one. I’ve reviewed AI-generated “error handling” tests that asserted an error toast appeared and ignored that the order had been placed anyway. In a QA environment, mildly embarrassing. In production, that’s a customer charged twice during a payment-provider incident.

Pattern 3: Assert what must NOT happen (excessive I/O, ~8x)

The most lopsided category in the CodeRabbit data is excessive I/O — around 8x. AI code loves redundant fetches, duplicate writes, and retry loops that weren’t asked for. No generated test will catch this, because the happy-path outcome is identical; the waste is invisible in the UI. Count the calls:

test('autosave fires exactly one request per edit', async ({ page }) => {
  const drafts: string[] = [];
  await page.route('**/api/drafts', async route => {
    drafts.push(route.request().method());
    await route.continue();
  });

  await page.goto('/editor/q2-report');
  await page.getByRole('textbox', { name: 'Body' }).fill('Q2 numbers are in.');
  await expect(page.getByText('Draft saved')).toBeVisible();

  expect(drafts).toEqual(['POST']);
});

toEqual(['POST']) asserts the count and the method in one line: three POSTs and a stray GET fail loudly. This is also the cheapest of the three patterns to retrofit: one route handler per suspicious endpoint.

Who Writes the Oracle?

My rule after two years of running AI-generated tests in real suites: the model that wrote the code doesn’t get to be the only author of its oracle. Scaffolding, locators, page objects, even the walk through the UI — generate all of it, and put a review workflow built for AI-generated tests around the output so checking it doesn’t become the new bottleneck. The assertion on any money path, any error path, and any side effect gets written by a human who read the spec instead of the code.

What I’d push back on hardest is the idea that better prompting fixes this. Asking for “edge cases and negative tests” gets you tests that look adversarial; all 4 of mine were wrong in ways that still passed. The model can’t know what makes a good test for your system, because the answer lives in your incident history, not in the diff.

Here’s the next step: open the newest AI-generated spec file in your repo and ask of each test, “would this still pass if the core calculation were inverted?” Every yes is a happy-path overfit. Take the three worst and rewrite them this week — one invariant, one forced failure, one call count. That’s about an hour of work against a defect class that’s measurably 1.7x more likely to be in your codebase than it was two years ago.