We Deleted 612 Redundant Tests and Caught More Bugs

The meeting lasted eleven minutes. I asked to delete 612 automated tests from a suite whose test count went into the QA director’s quarterly deck, and her first question was who would sign the risk acceptance. Seven weeks later the tests were gone, the nightly run was 72 minutes shorter, and escaped defects over the next two quarters fell from 11 to 4.

Here is how we decided which redundant tests to delete. Per-test coverage overlap gave us candidates, mutation testing showed a faster test already caught the same faults, and five written criteria got the list through an internal audit. Coverage alone would have deleted the wrong tests.

Why was a 2,140-test suite catching so little?

Because it was too slow to run where bugs are cheap. At 3 hours 10 minutes on a 12-node Selenium Grid, the suite could only run nightly. A regression merged at 10 AM was found the next morning, triaged by lunch, and fixed two days after it landed.

The client was a provincial crown corporation with a customer accounts and payments portal. The stack was Java 17, Selenium WebDriver and TestNG, plus 380 API tests that ran against the account service started in-process. Five SDETs and me. For six years the rule had been: every production bug gets a new UI test. Nobody ever removed one.

Each nightly run averaged 31 failures, about 70% unrelated to the product. People learned to skim the report, and a suite you skim is already missing bugs.

Dead ends: what didn’t find redundant tests

Duplicate names found 31 tests

Grouping tests by similar names and identical page-object call sequences found 31 copy-paste duplicates. Worth deleting, but 31 out of 2,140 doesn’t move a 3-hour run.

Coverage overlap alone deleted the wrong tests

Next we measured which application lines each test executed and flagged any test whose lines were fully contained in the union of the other tests. That produced 1,104 candidates, more than half the suite. It looked too good, so we piloted it: we archived 50 of those candidates on a branch and seeded 20 hand-written faults into a staging build.

Four of the 50 were the only tests that caught a seeded fault. One of them checked the tax rounding printed on the payment receipt. Three other tests ran the same rounding code and asserted the order total, never the receipt line. Running the same lines is not the same as checking the same result. Coverage shows a test was in the room, not whether it was watching.

How do you find redundant tests safely?

Use two signals together. Per-test coverage overlap gives you candidates. Mutation testing on your fast tests tells you whether the code those candidates exercise is already guarded below the UI. Then a human reads the assertions. Each signal alone is cheap and misleading. Together they gave us a list we could defend line by line.

Step 1: Per-test coverage from a running app server

The UI tests drove a deployed app, so coverage had to come from the server JVM. We started it with the JaCoCo agent in tcpserver mode. A TestNG listener then dumped and reset the counters after every test method:

public class CoverageDumpListener implements IInvokedMethodListener {
    // Coverage is per JVM, not per thread. Only valid at thread-count="1".
    private static final String HOST = System.getProperty("jacoco.host");

    @Override
    public void afterInvocation(IInvokedMethod method, ITestResult result) {
        if (!method.isTestMethod()) return;
        String testId = result.getTestClass().getRealClass().getSimpleName()
                + "." + result.getMethod().getMethodName();
        try {
            new ProcessBuilder("java", "-jar", "jacococli.jar", "dump",
                    "--address", HOST, "--port", "6300",
                    "--destfile", "target/cov/" + testId + ".exec",
                    "--reset") // zero counters so the next test starts clean
                    .inheritIO().start().waitFor();
        } catch (IOException | InterruptedException e) {
            throw new IllegalStateException("Coverage dump failed: " + testId, e);
        }
    }
}

The server side is one JVM flag. A jacococli report per .exec file then turns each test into an XML report of covered lines per source file, which a short script can compare:

# App server JVM
JAVA_OPTS="$JAVA_OPTS -javaagent:/opt/jacoco/jacocoagent.jar=output=tcpserver,address=*,port=6300"

# One report per test
for exec in target/cov/*.exec; do
  java -jar jacococli.jar report "$exec" \
    --classfiles app/build/classes --xml "${exec%.exec}.xml"
done

Serial is the catch. JaCoCo counts per JVM, so two tests running at once blur into one coverage file. The measurement run took 19 hours over a weekend at one thread. The first attempt was garbage anyway, because a scheduled statement-generation job hit the same app server at 2 AM and showed up as “coverage” in 140 tests. Use a dedicated environment with every batch job switched off.

Step 2: Mutation testing on the layer below

PIT mutates bytecode inside the JVM where the tests run, so it can’t mutate a server that a browser test reaches over HTTP. That constraint turned out to be useful. We ran PIT against the 380 in-process API tests, and for each class it told us whether a fast test already caught faults there:

mvn -pl account-service org.pitest:pitest-maven:mutationCoverage \
  -DtargetClasses='ca.portal.accounts.*' \
  -DtargetTests='ca.portal.accounts.api.*' \
  -DoutputFormats=XML,HTML \
  -Dthreads=6

The account service scored 61%. If a UI candidate touched a class with surviving mutants, it stayed, because it might be the only thing guarding that code. If every mutant in the classes it touched was killed by an API test, the UI test was re-checking logic that a 200-millisecond test already covered. Mutation testing usually exposes tests that assert nothing; here it proved which tests were safe to lose.

What were the deletion criteria?

A UI test was deleted only if it met all five of these. Each row shows how many candidates were left after that filter.

Criterion 3 is the one people skip. Two SDETs read every assertion over four days, and it rescued 46 tests the tools had marked redundant.

How do you get approval to delete automated tests?

You change what the organization measures before you change the suite. The analysis was finished in week three. The politics took four more weeks. I’d flip that order next time.

The KPI was the test count

“Automated test cases: 2,140” sat in the director’s quarterly deck as a line going up and to the right, and it rolled into her VP’s objectives. Deleting 612 tests meant presenting a 29% drop. We replaced the metric with three numbers she could defend upward: escaped defects, PR feedback time, and the nightly false-failure rate. Those are the same three I argue for in why pass rate and coverage mislead leadership. She took that deck to her VP herself, which is why the change stuck.

Audit wanted traceability, not test count

Internal audit didn’t care about test count. They cared that 143 requirement IDs in the trace matrix pointed only at candidates. We remapped each to a surviving test or a new API test, and the audit lead signed the mapping in two days.

The dev manager wanted a way back

His objection was fair: “You’re removing safety nets before our Q3 release.” So nothing died on day one. The 613 tests moved to an archived TestNG group that ran nightly in a shadow lane for 30 days without gating, and the archive had a hard 90-day delete date. I’ve seen a quarantine lane with no exit grow from 8 to 287 tests, so the expiry date was not up for negotiation.

The shadow lane earned its keep once. In week three it caught a French-language label regression on the payment confirmation page that the kept suite missed. Both language versions run the same Java code path and pull their text from resource bundles, so coverage saw the French test as a duplicate of the English one. We restored that test and added a rule: a locale variant stays unless another kept test asserts that locale. Final count: 613 archived, 1 restored, 612 deleted.

Results: fewer tests, more bugs caught

Deleting tests doesn’t catch bugs by itself. It freed up time. The 612 removed tests were mostly long multi-page journeys, so runtime dropped faster than test count. That freed Grid capacity for a 310-test critical-path gate on every PR, which had been impossible at 3 hours. We also spent about a third of the saved maintenance time writing 38 API tests aimed at the account service’s surviving mutants.

• Nightly runtime: 3h 10m to 1h 58m on the same 12 nodes
• False failures per night: 31 to 12, so people read the report again
• Account service mutation score: 61% to 78% after the 38 new API tests
• Restores after hard delete: zero

This wasn’t a controlled experiment, and two quarters is a small sample, so treat 11 to 4 as directional. The number I trust is the PR gate: 27 regressions found before merge instead of the next morning.

What I’d do differently

Start with the KPI conversation. I spent three weeks building analysis and almost lost it in eleven minutes. A test count in a deck is a commitment someone has made upward. Offer them a better number before you threaten the old one.

Key coverage by locale and feature flag from day one. The French test was nearly deleted because our coverage model only saw code paths. Data-driven variants (locale, province, account type) run the same lines and catch different bugs.

Use the word “delete”. A week of calling it “optimization” made people suspicious. “Delete, archived 90 days, shadow lane” built more trust.

Your next step

Pick your 50 slowest UI tests. Tonight, run them one at a time against a quiet environment with the JaCoCo agent in tcpserver mode and the dump listener above. If more than ten are coverage subsets, take that to whoever owns your test-count KPI before touching a file. If your team already budgets sprint time for flaky tests, redundant tests are the cheapest place to win that time back.

FAQ