A Self-Healing Locator Hid a Bug for 3 Weeks

For 23 days, one test in our regression suite was green every single night. For those same 23 days, passengers at a North American airline paid for a checked bag, watched it appear on their confirmation page, and then arrived at the counter to find no bag on their reservation. The test that existed specifically to prevent that had stopped clicking the bag button on day one — and told nobody.

That is a self healing test false positive, and it is the failure mode nobody quantifies. Every vendor page warns about it in a single hedged paragraph before pivoting to the product pitch. The academic work is blunter: a systematic review of 55 AI testing tools (Garousi et al., 2024) states it plainly — “If self-healing AI incorrectly replaces UI element locators, test cases may continue to pass while actually interacting with the wrong elements.” Fifty-seven percent of the tools they reviewed ship self-healing. None publish a false-heal rate.

Here is one incident, in full, with the numbers.

The setup

The suite: 640 Selenium tests in Java and TestNG, five SDETs, covering booking, check-in, and ancillaries on the airline’s web platform. Healenium wrapped the driver. It had been in place for fourteen months and the team liked it — the nightly wall of red after every design-system bump had genuinely stopped.

On March 4, the Extras step of the booking flow — where you add bags, seats, and sports equipment — was rebuilt on the company’s new component library. Every ancillary became a repeated row with a quantity stepper. The old markup had per-ancillary hooks. The new markup did not.

<li class="ancillary-row" data-ancillary="CHECKED_BAG">
  <span class="ancillary-row__label">Checked bag</span>
  <button class="stepper__inc" aria-label="Add"></button>
</li>
<li class="ancillary-row" data-ancillary="SPORTS_EQUIP">
  <span class="ancillary-row__label">Sports equipment</span>
  <button class="stepper__inc" aria-label="Add"></button>
</li>

Two sibling <li>s. Same classes. Same tag. Same accessible name. The only thing distinguishing the checked-bag stepper from the sports-equipment stepper is an attribute on the grandparent.

What does a self-healing false positive actually look like?

It looks like a heal that scores high, resolves fast, and binds to the wrong element — and then persists. Healenium catches the NoSuchElementException, runs its Longest Subsequence Comparison against the current DOM, ranks candidates, takes the top one, performs the action, and lets the test continue. There is no confirmation step in the open-source product.

Our locator was [data-qa='add-bag-pax-1']. It stopped existing on March 4. Healenium’s top candidate was the stepper__inc button in the sports equipment row — structurally near-identical to the one that vanished, one <li> further down. It scored 0.86.

The heal was written to Healenium’s Postgres backend and became the new reference locator. This is the part people miss: a bad heal is not a one-run event. It is a permanent substitution. Every night for the next three weeks, the test clicked “add sports equipment” without ever raising a warning, because from Healenium’s perspective there was nothing left to heal. The locator worked now.

If you want the mechanics under the hood, I’ve covered how self-healing locators actually work separately. What matters here is that the algorithm did exactly what it was designed to do. The algorithm was not the bug.

Why did the test still pass after healing to the wrong element?

Because the assertion was too coarse to tell the two paths apart. The test’s final check hit the reservation API and asserted the booking had some ancillary attached. A sports-equipment ancillary satisfies that just as well as a bag does.

@Test
public void checkedBagAppearsOnReservation() {
    extrasPage.addCheckedBag(PASSENGER_1);   // healed to the sports-equipment stepper
    String pnr = checkoutPage.payAndConfirm();

    Reservation reservation = reservationsApi.get(pnr);

    // The assertion that let 3,140 bookings through.
    assertThat(reservation.getAncillaries()).isNotEmpty();
}

I have no defence for line 9. Somebody wrote isNotEmpty() to avoid hardcoding an ancillary code, and it sat there for two years being harmlessly weak, because the only path that could reach it was the path that added a bag.

That is the actual lesson, and it is not the one the vendor blogs teach. The industry’s stock guardrail is “heal locators, never assertions.” We followed it. It didn’t matter. Self-healing does not need to touch your assertions to defeat them — it changes which path arrives at them. Every coarse assertion in your suite is safe only as long as your locators are honest about where they point. Turn on self-healing and you have quietly converted them all into live grenades. mabl’s own documentation concedes the same shape of failure: an “is present” assertion that auto-heals to a different element still passes.

This is the same disease as a green test suite hiding real bugs — except a human isn’t making the decision anymore, and it happens overnight, in CI, at machine speed.

The bug the test was supposed to catch

The March 4 release did two things. It broke the locator, and — in an unrelated component on the same branch — it broke bag persistence. A stale closure meant the bag quantity from the Extras step never made it into the booking payload. Passengers were charged at checkout, saw the bag on the client-rendered confirmation page, and got a PNR with no bag on it.

One release, two consequences. The heal absorbed the first, and by absorbing it, hid the second.

How was it finally caught?

Not by the suite. Not by anyone in engineering. It was caught by an airport operations lead at a hub station, on day 23, who emailed to ask why her gate agents had manually re-added ninety bags to reservations in a week.

The humiliating detail is what happened before that. The first support ticket landed on day 6 — a passenger charged twice for one bag. Support classified it as a duplicate-charge billing issue, refunded it, and closed it. Three more came in over the next fortnight. Each was refunded individually. For seventeen days, we were paying money to make the symptom go away, and the ticket taxonomy had no field for “the bag isn’t on the reservation.”

How do you trace a green test back to a heal event?

Slowly, and with more luck than I’d like to admit. A green test leaves no failure artifact — no screenshot, no trace, no stack. You are debugging an absence.

Dead end one: the code was innocent

git log on the page object showed no changes in eight months. addCheckedBag() still read exactly as written. The locator string was still [data-qa='add-bag-pax-1']. Nothing in the repository was wrong, which is precisely why two of us lost most of a day to it — we were diffing source when the divergence lived in a database.

Dead end two: the logs were gone

Healenium’s client-side surface is a WARN line. It logs that it failed to find an element, then that it’s attempting to heal. That’s it — there is no listener interface, no callback, no event you can subscribe to. (Someone has been asking for one since 2023.) Our CI archived JUnit XML and screenshots for failures, and discarded raw stdout on green runs. Twenty-three nights of heal warnings had been written and thrown away.

What actually worked: query the backend

Healenium persists every heal. The backend exposes it over REST, and this is the endpoint nobody in the ecosystem mentions:

curl -s "http://hlm-backend:7878/healenium/report/data" \
  | jq '.data[] | select(.declaringClass | contains("AddCheckedBagTest"))'

The record came back with failedLocatorValue, healedLocatorValue, score: 0.86, and a screenShotPath — a screenshot of the healed element, highlighted, sitting one row below where it should have been. Timestamped the night of March 4.

That screenshot had existed for three weeks. Nobody had opened the report. We audited all of it: 187 heals in fourteen months, and not one had ever been reviewed by a human. Of those 187, 22 had bound to a functionally different element than the one they replaced — a 12% wrong-heal rate on our own suite. Two of those 22 were still hiding live defects on the day we looked.

Is a green self-healing test a false positive or a false negative?

Pedantically, it’s a false negative: the test passed, the application was broken. But the self-healing corpus — vendors and practitioners alike — has settled on calling this a “false positive heal,” and I’m not going to fight it. Argue about the label and you’ll miss the property that matters: the tool resolved an ambiguity in your favour without telling you.

Because at the DOM level, an element that vanished because a designer refactored it and an element that vanished because a developer broke it are the same observation. Self-healing is a system that resolves that ambiguity by always assuming redesign. It is structurally biased toward keeping you green. That bias is the product.

The four guardrails we shipped

public class HealGateListener implements ISuiteListener {
    @Override
    public void onFinish(ISuite suite) {
        List<HealRecord> heals = healeniumClient.fetchReportData()
            .stream()
            .filter(h -> h.getTimestamp().isAfter(suite.getStartTime()))
            .toList();
        if (!heals.isEmpty()) {
            throw new HealGateException(heals.size() + " locator(s) healed. Review before merge.");
        }
    }
}

What I’d do differently

I would not have turned self-healing off. That’s the reflex, and I think it’s wrong. Over fourteen months, Healenium absorbed roughly 165 legitimate locator breaks that would otherwise have cost us triage mornings. The 22 bad ones were the price, and the price was only unaffordable because we couldn’t see them.

What I’d change is the ordering. We adopted self-healing to fix a pain we felt, and built the observability never. It should have been the reverse: build the heal ledger first, run the tool in shadow for a sprint, then let it act. Running a healer in shadow mode before you trust it would have surfaced our 12% wrong-heal rate in one week, on a branch, for free — instead of on day 23, in a room with the VP of Digital and someone from Finance holding a refund spreadsheet.

And I’d have argued harder against isNotEmpty() in code review two years earlier, when it was free.

Your one concrete next step: if you run Healenium, curl http://<your-hlm-backend>:7878/healenium/report/data right now and count the records. If that number is greater than zero and no human has ever reviewed them, you don’t have a self-healing test suite. You have a suite that has been quietly rewriting itself, and you have no idea what it says.