Your Green Test Suite Is Hiding Real Bugs

A green pipeline doesn’t mean your application works. It means your tests passed — and those are two very different things. I’ve watched teams ship critical bugs to production with a perfectly green CI dashboard, because someone along the way decided to make the test pass instead of fixing the actual problem.

This isn’t a theoretical concern. Over 10+ years in enterprise QA, I’ve seen five specific patterns that turn your test suite into a liar. Each one starts as a “quick fix” and ends with a bug in production that your tests were supposed to catch.

The Real Cost of a False Green

When a test genuinely catches a bug, you have a choice: investigate and fix the root cause, or make the test shut up. The second option is faster. It clears the pipeline. It unblocks the sprint. And it’s one of the most dangerous things you can do to a codebase.

Every false green erodes trust in a way that compounds. First, one test gets “fixed” by working around the symptom. Then two. Then twenty. Six months later, nobody on the team trusts the suite anymore, so they stop paying attention to failures altogether. I’ve seen this cycle play out at three different enterprise clients, and the recovery always takes longer than the shortcuts saved.

Here are the five patterns I’ve seen do the most damage.

1. Handling Alerts That Shouldn’t Exist

This one is subtle because it looks like good test automation practice. An unexpected alert dialog pops up during a test run. The test fails. A developer adds a dialog handler to dismiss it, the test goes green, and everyone moves on.

But here’s the question nobody asked: why is that alert appearing in the first place?

// The "fix" that hides the bug
@BeforeMethod
public void dismissUnexpectedAlerts() {
    driver.switchTo().alert().accept(); // Why is this alert here?
}

@Test
public void testUserLogin() {
    loginPage.login("testuser", "password123");
    assertTrue(dashboardPage.isLoaded());
}

At one client, we had a session timeout alert firing on login because a backend service was returning stale tokens. The test team added a global alert handler, and the test went green. Meanwhile, real users were seeing that same alert in production — a jarring popup telling them their session expired before it even started. It took three weeks for a customer support ticket to surface what the test had already detected and then deliberately silenced.

If your test encounters a dialog you didn’t expect, that dialog IS the bug. Don’t handle it — fail loudly and investigate. I wrote about the mechanics of alert handling in how Selenium’s alert handling crashed our parallel suite and the Playwright side in onceDialog vs onDialog patterns. Those posts cover how to handle dialogs correctly — this post is about when you shouldn’t handle them at all.

2. Removing Assertions to Make Tests Pass

This is the most indefensible pattern on this list, yet I’ve seen it happen at two different companies. A test fails because an assertion catches a real discrepancy. Instead of fixing the code or updating the expectation, someone deletes the assertion.

@Test
public void testCheckoutFlow() {
    cartPage.addItem("Widget Pro");
    cartPage.proceedToCheckout();
    checkoutPage.enterPaymentDetails(validCard);
    checkoutPage.submitOrder();

    // Someone deleted these because they "kept failing"
    // assertEquals(confirmationPage.getTotal(), "$49.99");
    // assertTrue(confirmationPage.hasOrderNumber());

    // All that's left:
    assertTrue(confirmationPage.isDisplayed());
}

This test now verifies exactly one thing: that a page rendered. It doesn’t check the order total. It doesn’t check that an order number was generated. It’s a navigation test pretending to be a checkout test. A pricing bug, a failed payment processor call, a missing order record — all invisible.

The worst part? In code review, a deleted assertion is just a removed line. It doesn’t trigger any alarm. Nobody writes a comment saying “I removed this because it was failing.” The context vanishes with the git blame.

How to Catch This

Make assertion count part of your test review process. If a test named testCheckoutFlow has zero assertions about checkout behavior, something is wrong. Some teams I’ve worked with add a simple rule: every test method must contain at least one assertion that validates business logic, not just page presence.

3. Bumping Retry Count Instead of Investigating

This is the most socially acceptable pattern on the list, and that’s what makes it dangerous. A test fails intermittently. Someone adds retries: 2. The test starts passing. Sprint velocity stays on track. Everyone’s happy.

But that test is still failing on the first attempt. Something is wrong — a race condition, a timing issue, a resource leak, an actual intermittent bug in the application. The retry just rolls the dice again and hopes for a better outcome.

export default defineConfig({
  // "Fixed" flaky tests by adding retries
  retries: 3, // Was 0, then 1, then 2, now 3...
  use: {
    baseURL: 'https://staging.example.com',
  },
});

I watched one team gradually increase their global retry count from 0 to 3 over six months. Each bump “fixed” a batch of flaky tests. By the end, nobody on the team knew which tests were actually stable. The bugs were still there — they just stopped hearing about them.

The Escalation Pattern

Watch for this timeline on your team:

1. Test fails intermittently → add 1 retry
2. Still flaky → bump to 2 retries
3. New tests start flaking → bump to 3 retries globally
4. Pipeline takes 40 minutes now → nobody touches retry config because “it works”
5. Real bugs ship to production → “but all tests passed”

If your retry count has only ever gone up, you have a systemic problem that retries are masking.

4. Try-Catch Blocks That Swallow Failures

This one is rampant in Java-based test frameworks. A test step throws an exception — maybe an element wasn’t found, a timeout occurred, or an API returned an error. Instead of letting the exception propagate and fail the test, someone wraps it in a try-catch with an empty catch block or a log statement that nobody reads.

@Test
public void testProfileUpdate() {
    profilePage.navigate();
    profilePage.updateDisplayName("New Name");
    try {
        assertEquals(profilePage.getDisplayName(), "New Name");
    } catch (AssertionError e) {
        logger.warn("Display name check skipped: " + e.getMessage());
    }
    // Test "passes" regardless of whether the update worked
}

This is an assertion that cannot fail. It goes through the motions of checking something, but the catch block guarantees a green result no matter what. I’ve seen this in frameworks where a “utility method” wraps all assertions in try-catch to “prevent test interruption.” The entire suite becomes a no-op — hundreds of tests executing steps without ever validating outcomes.

A Variation: The Soft Assert Trap

Soft assertions (collecting failures and reporting them at the end) are a legitimate pattern. But I’ve seen teams configure soft asserts and then forget to call assertAll() at the end of the test:

@Test
public void testOrderDetails() {
    SoftAssert soft = new SoftAssert();
    soft.assertEquals(orderPage.getStatus(), "Confirmed");
    soft.assertEquals(orderPage.getTotal(), "$49.99");
    soft.assertTrue(orderPage.hasTrackingNumber());
    // Missing: soft.assertAll(); — none of the above matter
}

Three assertions, zero enforcement. The test passes even if every single check fails.

5. Asserting on the Wrong Thing

This one is sneaky because the test has assertions — it just asserts something that can never meaningfully fail. The most common version: checking an HTTP status code without validating the response body.

@Test
public void testCreateUser() {
    Response response = apiClient.post("/users", newUserPayload);

    // This passes even when the API returns an error
    assertEquals(response.getStatusCode(), 200);
    // Never checks: response.body().path("id") is not null
    // Never checks: response.body().path("email") matches input
}

Many APIs return 200 with an error payload instead of using proper HTTP status codes. If your test only checks the status code, it’ll pass even when the API says {"error": "email_already_exists", "user": null}. The test is green. The user was never created.

I’ve seen this same pattern with UI tests that assert on page titles or generic success banners instead of the actual data:

@Test
public void testProductSearch() {
    searchPage.search("wireless keyboard");

    // This passes even if search returns wrong results
    assertTrue(resultsPage.hasResults()); // Just checks count > 0
    // Never checks: results actually contain "wireless keyboard"
}

The search could return completely unrelated products and this test would pass. It’s verifying that the search feature runs, not that it works correctly. As I covered in what makes a good test, a test that can’t fail for meaningful reasons isn’t protecting you from anything.

How to Audit Your Suite

If any of these patterns sound familiar, here’s a concrete audit you can run this week:

Search for Suppression Patterns

# Find empty catch blocks
grep -rn "catch.*{" --include="*.java" -A1 | grep -B1 "}"

# Find commented-out assertions
grep -rn "// .*assert" --include="*.java"

# Find try-catch around assertions
grep -rn "try" --include="*.java" -A3 | grep "assert"

# Check retry config
grep -rn "retries" --include="*.config.*"

Establish Team Rules

These three rules would have prevented every example in this post:

1. No global exception handlers or dialog dismissers — if something unexpected happens, the test fails
2. Every test must assert business logic — page presence alone is never sufficient
3. Every retry triggers an investigation — tracked in your issue tracker, not ignored

The point isn’t to ban retries or try-catch blocks. They have legitimate uses. The point is that each one should be a conscious decision with documented reasoning, not a reflexive shortcut to make the pipeline green.

Stop Making Tests Pass — Start Making Them Honest

A test suite that lies to you is worse than no test suite at all. At least with no tests, you know you’re flying blind. A false green gives you confidence where none is warranted — and that confidence ships bugs.

The next time a test fails, resist the urge to make it pass. Instead, ask: is this test telling me something important? Nine times out of ten, it is.

Your one action for today: pick one of the grep commands from the audit section above. Run it against your test codebase. If you find even one instance of these patterns, you’ve found a place where bugs are hiding behind a green checkmark.