What Makes a Test Flaky
A flaky test is one that passes and fails intermittently against the exact same code, with no changes to the logic under test. This nondeterminism destroys trust in the suite: engineers start re-running failed builds 'just in case' instead of investigating, and eventually a genuine regression gets waved through because 'that test is always flaky.' The root cause is almost always a hidden dependency on something the test doesn't fully control - wall-clock time, network latency, thread scheduling, shared global state, or the order other tests ran in.
Cricket analogy: It's like a DRS review that sometimes gives the same delivery as out and sometimes as not-out depending on which camera angle happened to be available that day - the ball didn't change, but an uncontrolled factor did.
Common Root Causes
Time-based flakiness happens when tests assert on wall-clock behavior - sleeping for a fixed duration and hoping an async operation finished, or comparing timestamps that can tie or roll over. Concurrency flakiness happens when tests share mutable state across threads, processes, or test cases without proper synchronization, so results depend on scheduling order. Order-dependency flakiness happens when one test leaves behind database rows, files, or in-memory singleton state that a later test unknowingly relies on or is corrupted by. External-dependency flakiness happens when a test hits a real network service, a real clock, or randomized data (like UUIDs or Faker-generated values) without seeding or mocking it.
Cricket analogy: It's like a chase target calculation under Duckworth-Lewis that gives a different revised score depending on exactly which minute rain interrupted play - a time-based dependency baked into the rule.
// FLAKY: races against a fixed sleep and shares module-level state
let cache = {};
test('loads user profile', async () => {
fetchUserAsync(1); // fire-and-forget, no await
await sleep(100); // hopes 100ms is enough
expect(cache[1].name).toBe('Ada');
});
// FIXED: awaits the actual completion signal and isolates state
test('loads user profile', async () => {
const cache = {};
await fetchUserAsync(1, cache); // awaited, deterministic completion
expect(cache[1].name).toBe('Ada');
});Diagnosing and Fixing Flaky Tests
The first diagnostic step is reproduction: run the suspect test in isolation and then in a tight loop (e.g. 'run 200 times, stop on first failure') to confirm it is genuinely nondeterministic rather than order-dependent. If it only fails alongside other tests, run it randomized and shuffled to find the polluting neighbor. Tools like pytest-randomly, JUnit's random test order, or CI matrix jobs with different seeds surface order-dependency bugs quickly. Once reproduced, bisect the cause: replace real clocks with fake/injectable clocks, replace real network calls with fakes or contract-tested stubs, and replace 'sleep and hope' with explicit wait-for-condition polling or awaiting a real completion signal.
Cricket analogy: It's like a third umpire re-running the same replay in slow motion from multiple angles until the exact frame where the bail dislodges is isolated - repeated, controlled reproduction beats guessing.
A useful CI hygiene practice: quarantine known-flaky tests into a separate job that reports failures but doesn't block merges, while tracking them in an issue with an owner and a deadline. Quarantine is a temporary triage tool, not a place tests should live forever - an unowned quarantine list rots into the same 'ignore all failures' problem it was meant to solve.
Retrying a failing test until it passes (test-level auto-retry) can mask real bugs, especially race conditions in production code that only manifest under load. Use retries sparingly, log every retry as a signal for investigation, and never let a test suite report 'green' without surfacing how many retries it took.
- A flaky test passes and fails on unchanged code because of a hidden, uncontrolled dependency.
- Common causes: wall-clock timing, concurrency races, shared/global state, test order dependency, and unmocked external services.
- Reproduce flakiness deliberately by running the test in a loop and with randomized test order before attempting a fix.
- Replace 'sleep and hope' with explicit wait-for-condition or awaited completion signals.
- Isolate shared state per test instead of relying on module-level or global singletons.
- Quarantine flaky tests with an owner and deadline rather than ignoring or silently retrying them forever.
- Mock or fake external dependencies (clocks, network, randomness) to make behavior deterministic.
Practice what you learned
1. What is the defining characteristic of a flaky test?
2. Which practice best diagnoses whether a test's flakiness is caused by order dependency?
3. Why is 'sleep for a fixed duration and then assert' considered a flakiness anti-pattern?
4. What is the primary risk of relying on automatic test retries to achieve a green build?
5. Which of the following is NOT a typical root cause of flaky tests?
Was this page helpful?
You May Also Like
Test Isolation and Independence
Understand why each test must run independently of every other test, and the patterns - fresh fixtures, dependency injection, and cleanup - that guarantee it.
Continuous Testing in CI/CD
Learn how to design a test pipeline - from fast unit tests to slower integration and end-to-end suites - that gives developers fast, reliable feedback inside a CI/CD workflow.
Arrange-Act-Assert Pattern
Master the three-phase structure - Arrange, Act, Assert - that keeps unit tests readable, focused, and easy to debug when they fail.
Related Reading
Related Study Notes in Software Engineering
Browse all study notesMicroservices Study Notes
Software Architecture · 30 topics
Software EngineeringDesign Patterns Study Notes
Software Design · 30 topics
Software EngineeringSoftware Engineering Study Notes
Python · 75 topics
Software EngineeringGit & Version Control Study Notes
Bash · 40 topics
Software EngineeringSystem Design Study Notes
Architecture · 46 topics