Flaky test
A flaky test is a test whose outcome varies across repeated executions under equivalent conditions without a relevant change to the software or test intent.
The Art of Unit Testing 3e
Test flakiness is the tendency of tests to produce inconsistent pass-or-fail results without a relevant code change.
- How it is used
- Identifies inconsistent test results so failures are investigated rather than dismissed or mistaken for production bugs.
- How deep it goes
- Defines the symptom, traces dependency-related causes, and gives a repeat-run check plus fix, convert, or kill remedies.
- Examples
- Uses Jest verifier setup, changing dates or random values, and a real network endpoint replaced by a stub.
- What you can do after
- Diagnose/debug with it by checking repeated runs, isolating inconsistent tests, and choosing whether to fix, convert, or remove them.
Vibe Engineering
A flaky test is an automated test that intermittently passes or fails without a corresponding change in the behavior being tested.
- How it is used
- Treats test reliability as a prerequisite for trustworthy agent CI feedback and faster development.
- How deep it goes
- Details rerun tracking, quarantine, and incremental repair, with timing dependencies and mock races as examples.
- Examples
- Uses a 2,400-test monorepo and twelve high-rerun tests to illustrate agent iteration waste and quarantine.
- What you can do after
- Diagnose unreliable CI feedback and prioritize quarantine or repairs using rerun data.
Build High Value Software Systems
A test whose result is nondeterministic despite unchanged software and test inputs.
- How it is used
- Frames flaky tests as flow-blocking defects that make build results unreliable and encourage reruns or ignored failures.
- How deep it goes
- Defines nondeterminism under unchanged code and environment, then traces workflow costs; stops short of a general repair procedure.
- Examples
- Uses a time-coupled order-expiration test, an asynchronous user insert, and a flickering-warning analogy.
- What you can do after
- Diagnose likely timing or synchronization causes and question whether retries conceal symptoms instead of removing causes.
Read how Build High Value Software Systems explains flaky test →
100 Java Mistakes and How to Avoid Them
A flaky test is an automated test whose result varies between runs despite unchanged code and inputs.
- How it is used
- Uses flaky tests to explain how unstable external state undermines continuous integration.
- How deep it goes
- Explains timing and filesystem race conditions with focused Java snippets, then stops at practical avoidance guidance.
- Examples
- Uses a backup-folder timestamp assertion and filesystem file-size checks to demonstrate changing state between operations.
- What you can do after
- Diagnose timing and state-change failures, then stabilize tests by reusing observations and reducing separate filesystem queries.
Read how 100 Java Mistakes and How to Avoid Them explains flaky test →
Also covered in
- 100 Go Mistakes and How to Avoid Them
- Infrastructure as Code, Patterns and Practices: With examples in Python and Terraform
- Go by Example
- Contract Testing in Action
What to know first
- integration testAn integration test evaluates whether multiple software components, or a component and an external dependency, interact and behave correctly together.
- software testingSoftware testing is the systematic evaluation of software under specified inputs or conditions to assess its behavior against expected results or requirements and identify defects.
- assertionA software assertion is an executable check of an expected condition or property that signals failure when the check does not hold.
- bugA software bug is a defect that can cause software to behave incorrectly or differently from its intended or specified behavior.
Related concepts
- bugkind of · A software bug is a defect that can cause software to behave incorrectly or differently from its intended or specified behavior.
- provider stateaddressed by · A provider state is a named precondition describing the provider data or runtime situation required before an interaction is verified.
- trustcontrasts with · Trust is confidence that an entity or source is reliable and appropriate to rely on in a given context.
- Andon cordcontrasts with · A cord that a worker pulls to signal an abnormality and prompt immediate attention or correction.
- automated testingkind of · Automated testing is the execution and evaluation of software tests by programs or tools rather than by manually performing each test procedure.
Questions about flaky test
What is flaky test?
A flaky test is a test whose outcome varies across repeated executions under equivalent conditions without a relevant change to the software or test intent.
Which Manning books explain flaky test?
The Art of Unit Testing 3e, Vibe Engineering, Build High Value Software Systems and 100 Java Mistakes and How to Avoid Them.
What should I know before learning flaky test?
Manning books that teach flaky test usually introduce integration test, software testing, assertion and bug first.
What is the difference between flaky test and trust?
A flaky test is a test whose outcome varies across repeated executions under equivalent conditions without a relevant change to the software or test intent. Trust is confidence that an entity or source is reliable and appropriate to rely on in a given context.
What is flaky test a kind of?
Flaky test is a kind of bug. A software bug is a defect that can cause software to behave incorrectly or differently from its intended or specified behavior.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



