The SDET Bus Factor Nobody Talks About

Our test framework had nine contributors in git blame and exactly one person who could explain why thread-count was set to 5. That’s a bus factor of 1 in test automation, sitting inside a suite that looked perfectly healthy. When he took a month of parental leave, the other eight of us found out we’d been writing tests on top of a machine none of us could repair.

Every enterprise team tracks bus factor for the payments service and the auth service. Nobody tracks it for the test framework, because the framework works — green most mornings, and its riskiest code untouched in eighteen months. That looks like health. It’s the definition of the problem: the parallel-execution and thread-safety layer is the one part of the codebase nobody has to understand in order to do their job, right up until the day everyone does.

What is the bus factor in a test automation framework?

Bus factor is how many people would have to get hit by a bus before a project stalls. For a test framework the honest number is almost always 1, and it’s 1 for a specific layer rather than the repo as a whole. Everyone can add a test. Exactly one person can tell you what happens when you change the thread pool.

Avelino et al. computed truck factors across 133 popular GitHub projects and found 65% have a truck factor of 2 or less, with 34% sitting at exactly 1. Worth noting which project they name at TF=2: cucumber/cucumber. The testing tool your framework is probably built on had a truck factor of two.

How a one-line thread-count change froze our pipeline for six days

At a large Canadian enterprise, we had a Java/TestNG/Selenium Grid suite: 1,400 tests, nine contributors, a Grid provisioned for 40 concurrent sessions. One engineer — the framework owner, whether or not anyone had written it down — built the driver management layer three years earlier and had maintained it alone since.

He went on leave. Two weeks in, someone opened a perfectly reasonable PR to speed up the nightly.

The change that looked safe

<suite name="regression" parallel="methods" thread-count="12">
  <!-- was thread-count="5" — bumping to cut the nightly from 90m -->
  <test name="checkout">
    <classes>
      <class name="com.example.tests.CheckoutTests"/>
    </classes>
  </test>
</suite>

Two people approved it. It looks like arithmetic: more threads, less wall-clock. What none of the three of us knew — and I include myself, I approved it — is that thread-count is not the only thread pool in a TestNG suite. It’s one of three.

A dozen of our data-driven classes fed 8-12 scenarios through a parallel data provider:

@DataProvider(name = "checkoutScenarios", parallel = true)
public Object[][] checkoutScenarios() {
    // 11 payment scenarios — each one claims its own browser,
    // from a pool sized by data-provider-thread-count, not thread-count
    return new Object[][] {
        { "visa-credit", 49.99 },
        { "interac-debit", 120.00 },
        // ...9 more
    };
}

TestNG’s data-provider-thread-count defaults to 10 and is a separate pool from thread-count. The two multiply. Do the arithmetic nobody did:

BEFORE  thread-count=5   ×  DP pool 10  →  up to  50 sessions
AFTER   thread-count=12  ×  DP pool 10  →  up to 120 sessions
GRID CAPACITY                           →        40 sessions

At 5, only a handful of data-driven classes ever overlapped, so peak demand hovered in the mid-thirties and the suite squeaked under the ceiling every night. We had been running one config change away from a cliff, and nobody in the building knew where the cliff was. At 12 we went over it on the first run: the Grid started refusing sessions with SessionNotCreatedException, and the nightly went red.

Why the revert didn’t fix it

Here’s the part that turned a bad afternoon into a bad week. We reverted the PR on day two. The pipeline stayed broken.

The two pools don’t just multiply — they hand work to different threads, and our teardown assumed they didn’t:

@AfterMethod
public void tearDown(ITestResult result) {
    // The driver was created on a data-provider pool thread.
    // @AfterMethod runs on a different one — so this ThreadLocal
    // lookup misses, returns null, and we bail out before quit().
    WebDriver driver = DriverFactory.getDriver();
    if (driver == null) return;
    driver.quit();
    DriverFactory.remove();
}

A ThreadLocal is per-thread. Once the data-provider pool saturated, drivers were being created on threads that @AfterMethod never ran on — the lookup missed, getDriver() returned null, we returned early, and a live browser sat on a Grid node with nobody holding its reference.

Normally the Grid reaps those: a node kills any session idle longer than --session-timeout, default 300 seconds. Ours was set to 1800, because years earlier somebody had raised it to stop long reconciliation tests being killed mid-run. So orphans lingered half an hour — and the reaper only clears the session. The chromedriver and chrome processes it spawned stayed resident, accumulating across runs until the nodes were starved.

That’s why the revert did nothing. The bad config was gone; the wreckage was on machines we weren’t looking at. Eight engineers stared at SessionNotCreatedException for four more days, unable to tell whether they were seeing the original bug, a new one, or the kind of race condition that only looks like flakiness. Two releases slipped.

On day six someone gave up and phoned the framework owner on his leave. He asked one question — “did anyone drain the nodes?” — and diagnosed it in about twenty minutes.

Six days for eight engineers. Twenty minutes for the one who understood the machine. The failure wasn’t hard. Understanding it was, and understanding was the thing we’d centralized in a single human being.

Why does only one person understand the concurrency layer?

Because it’s the only layer nobody needs in order to do their job. Adding a test teaches you the test API. Adding a page object teaches you the locator strategy. Neither teaches you why DriverFactory wraps a ThreadLocal, or how many sessions the Grid can actually serve. That knowledge has no daily on-ramp — you get it by building the thing, or by an outage.

Git blame lies about which layer is at risk

Run bus-factor tooling on a test repo and it’ll report something reassuring, because it counts files. Our nine contributors were real. Here’s what they’d actually touched:

Repo-level bus factor said 9. The number that mattered was 1, and it was sitting in the two paths that took the pipeline down.

The ICSE-SEIP 2022 study of 269 engineers found that VCS-only bus factor tools miss the knowledge channels that matter most — code review, pairing, the conversations where understanding actually transfers. Compute bus factor per layer, not per repo. Point it at driver management, parallel config, and fixture teardown. That’s a fifteen-minute exercise and it’ll tell you something your dashboard won’t.

There’s a second tell, and it’s free: watch your retry annotations. When the person who understands the concurrency model is gone, the team’s only remaining move against a race condition is to suppress it. Retry counts creeping upward is a leading indicator of knowledge loss, not of flaky infrastructure — a green suite that’s hiding real bugs usually got that way because nobody left could diagnose the red one.

Sometimes you really do need the clever code

The strongest version of the counterargument: concurrency is hard, some of it is necessary, and “just make it simpler” is what people say when they’ve never had to make 1,400 tests finish inside a release window. A junior SDET should not be rewriting your thread pool. Some domains — high-volume Grid orchestration, custom sharding — need code a mid-level engineer can’t safely modify in their second week. Pretending otherwise is how you get a framework that’s legible and also takes four hours.

There’s a fair rejoinder from the research, too: a 2024 study of 36,000+ GitHub projects found that while 89.65% eventually lost all their core developers, 27% of those projects recovered by attracting new maintainers. Projects do survive their heroes leaving. But they survive because strangers adopt them — and no community of volunteers is waiting to pick up your company’s internal DriverFactory. Open source has a replacement pool. Your test framework has a hiring req, a three-month ramp, and a suite that’s red right now.

So the concession is real but narrow: keep the clever code where it earns its keep, then carry every line of it as a liability you’ve chosen deliberately, with a named second reader.

How do you reduce test framework knowledge silo risk?

Three moves: rotate a second engineer through ownership of the framework internals for a quarter, write runbooks for the three failure modes that actually recur, and design the concurrency layer to be boring on purpose. Each one reduces test framework knowledge silo risk by making the dangerous layer something more than one person has touched under pressure.

What doesn’t work is the advice every other bus-factor article lands on. Our framework had a README with an accurate section called “Parallel Execution.” It didn’t save us a single one of those six days, because a document describing a thread pool doesn’t transfer the ability to reason about one. The best statement of this I’ve seen is an engineer on Hacker News describing a departure that cost his team eight weeks: the leaver had explained where everything was, and “the other person simply didn’t understand the details.” Docs transfer location. They don’t transfer comprehension.

Rotate the framework owner for a quarter

Take someone who has only ever written tests and give them the framework internals for three months — with the previous owner still in the building, on call for questions, and explicitly barred from just fixing things himself. At the end of the quarter, two people have debugged the driver layer under real pressure. That’s the deliverable, not whatever code they ship.

This is expensive. That’s the trade: you’re buying down a risk that otherwise gets collected at the worst possible moment, at a price you don’t negotiate.

Write runbooks for the three failures that actually happen

Not a design doc — a runbook, for the three failure modes that recur in every Selenium-Grid-shaped suite I’ve worked on:

1. Session exhaustion. Symptom: SessionNotCreatedException, “Could not start a new session.” First check: the capacity math — thread-count × the data-provider pool, against the Grid’s real session ceiling.
2. Orphaned sessions and leaked browser processes. Symptom: the suite fails from the first test, and fails the same way after a revert. First check: the node’s --session-timeout, then drain and recycle the nodes before rerunning.
3. Silent serialization. Symptom: the suite went green and got much slower, and races stopped reproducing. First check: the parallel attribute. TestNG accepts methods, classes, tests, instances, and none — anything else silently falls through to NONE. Typo it as parallel="method" and your suite runs serially, forever, and nothing in the report tells you.

Each entry is symptom → first check → who to call. That’s a page. A team that has this page loses hours, not days.

Make the framework boring on purpose

The opinionated part, and the one I’ll defend: cleverness in a test framework is a bus-factor tax, and most enterprise suites should trade a little performance for a lot of legibility.

Be precise about what that means, because it cuts against advice I’ve given before — and against code I published. The ThreadLocal pattern that keeps drivers from leaking between threads isn’t the problem. A plain ThreadLocal<WebDriver>, set in @BeforeMethod and removed in @AfterMethod, is the boring choice, and for an existing TestNG suite I’d still make it the default on day one.

But the quitDriver() I published in that post guards on null and returns — the same shape as the teardown above, with the same hole. Boring code still has to handle the failure path, and mine didn’t. Legible isn’t the same as correct; it only means someone other than me could have caught it.

The tax is what teams build on top of the primitive: the driver registry using thread ID as a lookup key for shared state, the IExecutionListener that reflectively cleans up, the context propagation that exists because someone wanted @DataProvider threads to see the parent thread’s driver. Each is a line only its author can debug at 2am.

Two trades I’d make every time:

• Use ThreadGuard.protect(). One line, no overhead worth measuring, and it converts a three-day phantom-failure hunt into a stack trace that names the problem — Selenium throws immediately if a driver is touched from a thread other than the one that created it. It’s a detector, not a cure; the Selenium docs are explicit that it doesn’t replace ThreadLocal.

  WebDriver driver = ThreadGuard.protect(new RemoteWebDriver(gridUrl, options));

• On greenfield, delete the category instead of managing it. Playwright’s workers are OS processes, not threads — separate memory, nothing to share, no ThreadLocal, no ThreadGuard, no thread-affinity bugs. Process overhead is a real cost. But a framework whose concurrency story is “the OS handles isolation” has a structurally higher bus factor than one carrying a thousand lines of your own. I’ve argued for process-level parallelism over a ThreadLocal refactor on performance grounds; the bus-factor argument is the stronger one.

The rule: greenfield, pick the process model and skip the category entirely. Existing TestNG suite, plain ThreadLocal and nothing clever on top of it. Both are boring on purpose, which is the point.

What I’d do differently

I’d stop treating “the framework works” as the end of the conversation. For three years that suite’s health was measured entirely by whether it was green — and by that metric, the concurrency layer was our best-performing code right up until it was our most expensive. Tracking framework ownership as a standing risk belongs to whoever leads QA strategy on the team, and on that project nobody had claimed it.

I’d also own my part. I approved that PR. I’d been on the project two years, written a few hundred of those tests, and I could not have told you that data-provider-thread-count was a separate pool. That’s not a gap I was hiding; it’s one I didn’t know I had, because nothing in my daily work ever put me near it. Which is the whole argument: the silo isn’t maintained by an expert hoarding knowledge. It’s maintained by everyone else never needing it.

Do this today: run git log against only your driver management and parallel config paths, and count the distinct authors.

git log --format='%an' -- \
  'src/test/java/**/factory/*' \
  'src/test/java/**/base/*' \
  'src/test/resources/*.xml' \
  | sort | uniq -c | sort -rn

If that list has one name on it, you don’t have a framework. You have a person, and a lot of code that depends on him being reachable.

Frequently asked questions