A flaky test gives different pass-or-fail results even though the production code has not changed. It may fail, pass on a rerun, and fail again later. That makes a failure hard to interpret: it might reveal a real bug, or it might be noise from the test itself or its environment. A trustworthy test should fail for a meaningful reason. section 7.2.5 section 7.2
Flakiness is not the same as an ordinary failing test. A real production bug is a useful failure. A buggy, outdated, conflicting, or flaky test is untrustworthy until investigated. Calling every failure “flaky” is especially risky because it gives real defects an easy excuse to be ignored. section 7.2 section 7.4.3
Shared state can make parallel tests unreliable
Suppose tests share one verifier instance or an errors array outside their setup. One test can change that shared state while another test runs, so the outcome depends on timing. Create a fresh verifier and fake rule for each test instead. In Jest, parallel execution makes shared upper-scope state particularly hazardous. section 2.6 section 2.6
External and changing inputs are common causes. Current time, random numbers, machine names, databases, networks, configuration, third-party services, and security layers add moving parts. The more real dependencies a test uses, the less control it has over its execution path and the greater its chance of inconsistent results. section 7.4.5 section 7.5
Figure 7.1 The higher the level of the tests, the more real dependencies they use, which gives us confidence in the overall system correctness but results in more flakiness.
Compare test levels by their dependency trade-off: higher-level tests use more real dependencies, increasing overall confidence but also the opportunity for flakiness. section 7.5
When a test is confirmed flaky, reduce its cost deliberately:
- Define flakiness consistently, such as running the suite repeatedly without code changes and finding tests with inconsistent outcomes. section 7.5.1
- Quarantine it in a separate category or pipeline so it does not add noise to regular delivery feedback. section 7.5.1 section 7.6
- Fix it by controlling dependencies—for example, insert required database data as part of the test. section 7.5.1
- Convert it to a lower-level test by replacing a real dependency, such as a network endpoint, with a stub. section 7.5.1 section 3.8
- Kill it when its value does not justify its maintenance cost, especially when better tests already cover the behavior. section 7.5.1
Do not respond by eliminating integration or end-to-end tests altogether. They can provide confidence that lower-level tests cannot. Instead, keep a fast, nonflaky “safe green” area for frequent local feedback, and run higher-level or quarantined checks separately in automated pipelines. A balanced test strategy accepts the trade-off between confidence, speed, maintainability, complexity, and flakiness. section 7.4.3 section 10.5
