3 AI Testing Tools Hid the Bugs I Planted (2026)

Three tools. One application. 128 broken locators, and six real bugs I planted on purpose. This is the AI testing tool evaluation I ran for an enterprise client, and every tool cleared a 70% heal rate — the number that goes on the vendor slide. Every tool also turned at least two of my planted bugs green, and none of them told me it had done it.

That middle number doesn’t exist anywhere. I went looking before I ran this: vendor pages, the ranked listicles, Gartner’s first Magic Quadrant for AI-augmented testing tools (October 2025). They all report how often a heal succeeded. None report how often a heal was wrong.

Why every “best AI testing tools” list fails you

Because “success rate” is the one metric a false heal inflates. A heal that binds your locator to the wrong element also turns the test green — so the worse a tool’s judgment, the better its headline number looks.

Look at what the category claims. mabl advertises up to 95% of test maintenance eliminated. Functionize cites 99.9% healing accuracy. Virtuoso reports 83% — though that one’s a maintenance-reduction figure, not a healing figure, which is its own tell. Different apps, different suites, no denominators. They’re marketing numbers wearing a lab coat.

I’ve argued the success-rate-versus-precision case before, in why you should run a healer in shadow mode first. What that post couldn’t give you was a number across vendors. This one can — because I stopped reading and ran the trial.

Bias disclosure. I’ve spent far more of the last decade in Playwright and Selenium than in low-code platforms, and I built the measurement harness for Playwright before anything else, so Playwright’s instrumentation here was richer than mabl’s. I have no commercial relationship with any of the three vendors.

How do you evaluate AI testing tools without a six-month bake-off?

You build a fixed corpus of known breakages, plant bugs inside it that the tool is supposed to catch, and count how many it hides. Three weeks, and the only expensive part is the setup.

The app was the policy-administration platform at a large Canadian insurer — the same system where I’d built that shadow harness. This trial is that harness pointed at three vendors instead of one.

The corpus: 40 real UI changes, 6 planted regressions

I pulled 40 UI-affecting commits from the product’s previous nine months — design-system migrations, data-testid renames, a nav restructure, a modal that became a drawer. Real changes that had really broken tests. Replaying them against a frozen suite produced 128 broken locators per harness.

Then I did the part nobody does. I planted six functional regressions among them:

• The “Submit Policy” button disabled by a permissions regression, with an enabled “Save Draft” button sitting right next to it.
• The premium total rendering a stale value from the previous quote.
• A required beneficiary field that silently stopped persisting.
• Three more in the same spirit.

A correct tool goes red on all six. A heal on a planted regression is not a heal. It’s a masked bug.

The attach problem: does the tool even fit the suite you own?

Healenium wraps a Java Selenium WebDriver. Playwright’s Healer targets the TypeScript Playwright runner. mabl runs tests on its own cloud runner. To get a genuine head-to-head I built the same 60 user journeys three times, in three stacks, against one app — nine engineer-days, which dwarfs every licence fee in this post.

mabl does ship an importer, and I want to be precise because this is exactly the checkmark that decides a purchase. mabl tests import playwright exists, it’s labelled experimental, and mabl’s own docs state that variables, code blocks, if statements, and loops aren’t migrated. It imports the clicks and drops the logic. Worse for our purposes: imported element interactions land as custom find steps, and auto-heal only applies to converted mabl steps — so the parts that came across untouched were the parts that got no healing at all.

That’s the honest first question in any AI testing tool evaluation, and it isn’t “which tool heals best.” It’s: does this tool attach to the suite I already own, or does it want to rewrite it? “Imports Playwright tests: ✅” is technically true and practically meaningless.

What each tool does under the hood

Three different architectures — Healenium does DOM-tree similarity scoring, Playwright’s Healer does agentic LLM repair, mabl does a visual-plus-DOM hybrid. Those are three rows of the taxonomy in how self-healing locators actually work; I’m placing each tool in it, not re-explaining it.

The wiring tells you most of what you need to know about the commitment each one demands:

WebDriver delegate = new ChromeDriver(options);

// Healenium wraps the driver you already have.
// Zero changes to any test or page object.
WebDriver driver = SelfHealingDriver.create(delegate);

return driver;

# Playwright's agents generate into your repo,
# then run inside your existing LLM loop.
npx playwright init-agents --loop=claude

# mabl's path in, for comparison — note the flag:
mabl tests import playwright --project chromium

Healenium is a one-line driver wrap — non-invasive enough that a Java shop with a decade-old Selenium suite can trial it in an afternoon. Playwright’s Healer generates agent definitions into your repo. mabl’s import command is one line too, and that line is the beginning of a migration, not the end of one.

The dial nobody turns

Healenium ships a config file, and one line in it does more for your false-heal rate than any feature in this post:

recovery-tries = 1
score-cap = .6
heal-enabled = true
hlm.server.url = http://localhost:7878
hlm.imitator.url = http://localhost:8000

score-cap is the similarity threshold required to accept a heal, and it ships at 0.6. A 60% structural match to the element you last saw is enough for Healenium to click something and call it your button. Nearly every tutorial copies the default and never mentions it.

Does self-healing test automation actually work?

Yes — at turning red tests green, at rates between 70% and 87%. What doesn’t hold is the implied second half: that a green test is a working test. Measured against a known ground truth, self-healing test automation accuracy landed near seven correct heals in eight — meaning roughly one heal in eight bound to an element I wouldn’t have chosen.

Same app, same 128 broken locators, same six planted bugs:

Read the heal-rate row and mabl wins. Read the bold row and the ranking inverts. mabl healed the most and masked half my planted bugs. Playwright’s Healer — the tool I like most and use most — was the worst offender here, and I didn’t enjoy writing that sentence.

Why do self-healing tools bind to the wrong element?

Because they all optimize for resemblance, and a regression usually leaves a near-identical element sitting right next to the broken one. The disabled “Submit Policy” button with an enabled “Save Draft” beside it isn’t an adversarial edge case I invented to embarrass anyone. It’s what a permissions bug looks like in a real insurance UI.

Here’s what two of the tools did to that single planted bug:

FAILED:  By.cssSelector("#submit-policy")
HEALED:  By.cssSelector("#save-draft")
Score:   0.71          # cap is .6 — accepted
Result:  PASSED        # clicked the wrong button

test('quote shows correct premium', async ({ page }) => {
  await page.getByTestId('recalculate').click();
  await expect(page.getByTestId('premium')).toHaveText('$1,240.00');
  await expect(page.getByTestId('premium')).toBeVisible();
});

Healenium’s failure is mechanical: same tag, same parent chain, same sibling context, one word different in the label. It scored 0.71, cleared the shipped cap of 0.6, and clicked it. Playwright’s Healer did something worse — on two of the six it didn’t touch the locator at all, it weakened the assertion, downgrading a toHaveText to a toBeVisible until the bug fit through. That’s the failure mode I’ve covered in why the Healer agent hides your race conditions. Watching it do that to a bug I’d planted myself was a different experience than reading about it.

mabl healed onto the wrong button too. I can’t tell you precisely why — its heal decisions aren’t inspectable from outside, which is itself the finding.

I’m not the first to find this — it’s in the issue tracker

Healenium’s own repository contains this experiment, filed as a bug. In issue #88, a user comments out a Submit button, Healenium heals to the neighbouring Reset button, and isDisplayed() returns true. His words: “it can lead to false positives because the test passes even though the expected element is not actually present or interactable.” The healing report in that issue logs a score of 0.786 — comfortably over the shipped 0.6 cap, and comfortably under the 0.85 I’m about to recommend.

In issue #35, a renamed #submit causes a click on #submitAll — wrong business action, green build. Open since December 2023. Zero comments.

The skip channel nobody talks about

Playwright’s Healer has a second masking path, and it’s in the docs in plain sight. The agent’s output is specified as: “A passing test, or a skipped test if the healer believes that functionality is broken.”

Read that again. When the agent decides the feature is genuinely broken, it marks the test skipped — the right engineering instinct and a catastrophic CI outcome, because a skipped test doesn’t fail a build. Your pipeline exits zero. Your dashboard is green. The agent correctly caught a real regression and filed it somewhere nobody looks. Seven tests went this way, two of them my planted bugs.

The fix: one line of config

I raised Healenium’s score-cap from the shipped 0.6 to 0.85 and re-ran the entire corpus.

Heal rate fell 18 points. False heals fell by 75%, and every planted bug went red. That’s the trade, stated plainly: you buy safety with heal rate, and the default ships tuned for the demo, not for your suite. A tool that heals 52% of your locators correctly and fails loudly on the rest is worth more than one that heals 70% and lies to you about six of them.

Nobody’s going to turn this dial for you. It isn’t in the quickstart, and the vendors have no incentive to mention that their headline number drops 18 points when you make it trustworthy.

Which AI testing tool should an enterprise buy?

There’s no winner here, and anyone handing you one is selling something. The deciding variable is almost never heal rate — it’s what you already own.

Healenium is free, self-hosted, and its heal decisions are inspectable in Postgres — the only reason I could measure its false-heal rate at all. That auditability makes it the right answer in regulated environments, where “the tool rewrote the assertion and we don’t know why” isn’t a sentence you can put in front of an auditor.

Playwright’s Healer is worth running behind those two guardrails and reckless without them. It costs tokens instead of a contract, and its output is a patch — which means your PR review is the gate, if you actually review.

Buy mabl when you’re starting from nothing. Then the migration isn’t a cost, it’s the product, and mabl’s healing was the strongest of the three. Don’t buy it to rescue an existing suite — you’ll pay a migration, a subscription, and lock-in to inherit a 92% precision rate a config file could have given you.

The strongest argument against this whole category

Let me steel-man the position that beat me up in review. Several practitioners I respect argue that broken locators were never the real maintenance cost — that test data, environments, iframes, and shadow DOM eat the week, and locator churn is a rounding error you’re now paying a subscription to paper over. On this insurer’s suite, locator breakage caused roughly 15-20% of red tests. The other 80% were stateful, concurrent, environment-shaped failures no healer in this table can even see.

If that’s your distribution, the tool evaluation you’re about to run is optimizing the wrong 20%. The sharpest version came from a QA lead on Reddit and I still can’t argue past it: ask a developer why they don’t want self-healing code in their production application. Test code is still code.

What I’d do differently

Six planted regressions was too few. The gap between masking two and masking four is four bugs, and I don’t have the statistical footing to say Playwright’s Healer is meaningfully worse than Healenium — only that all three are worse than the marketing implies. Next time I’d plant thirty and get error bars instead of a ranking.

I’d also stop treating “did it heal correctly” as a question about locators. The most alarming result wasn’t a wrong locator — it was a gutted assertion. A false-heal rate measured only on locators would have scored that as a success, and I nearly built the harness that way.

And I’d have run the score-cap experiment in week one instead of week three. I spent two weeks measuring three vendors against each other when the largest single effect in the dataset was one number in a properties file that ships wrong.

Your one concrete next step

Plant one bug. This week, take the most business-critical flow in your suite, break it for real — disable the button, stale the total, drop the field — and leave a plausible sibling element next to it. Then run whatever healing you already have switched on, and watch what colour the build goes.

If it goes green, you’ve learned more about your tooling in an afternoon than any listicle will tell you. For the fuller build — the ledger schema, the assertion-weakening diff gate, the skip-count guard — start with the shadow-mode harness and point it at whatever you’re about to sign for.