liveBook · Manning

Flaky test

A flaky test is a test whose outcome varies across repeated executions under equivalent conditions without a relevant change to the software or test intent.

How 4 Manning books explain it

The Art of Unit Testing 3e

Test flakiness is the tendency of tests to produce inconsistent pass-or-fail results without a relevant code change.

How it is used
Identifies inconsistent test results so failures are investigated rather than dismissed or mistaken for production bugs.
How deep it goes
Defines the symptom, traces dependency-related causes, and gives a repeat-run check plus fix, convert, or kill remedies.
Examples
Uses Jest verifier setup, changing dates or random values, and a real network endpoint replaced by a stub.
What you can do after
Diagnose/debug with it by checking repeated runs, isolating inconsistent tests, and choosing whether to fix, convert, or remove them.

Read how The Art of Unit Testing 3e explains flaky test →

Vibe Engineering

A flaky test is an automated test that intermittently passes or fails without a corresponding change in the behavior being tested.

How it is used
Treats test reliability as a prerequisite for trustworthy agent CI feedback and faster development.
How deep it goes
Details rerun tracking, quarantine, and incremental repair, with timing dependencies and mock races as examples.
Examples
Uses a 2,400-test monorepo and twelve high-rerun tests to illustrate agent iteration waste and quarantine.
What you can do after
Diagnose unreliable CI feedback and prioritize quarantine or repairs using rerun data.

Read how Vibe Engineering explains flaky test →

Build High Value Software Systems

A test whose result is nondeterministic despite unchanged software and test inputs.

How it is used
Frames flaky tests as flow-blocking defects that make build results unreliable and encourage reruns or ignored failures.
How deep it goes
Defines nondeterminism under unchanged code and environment, then traces workflow costs; stops short of a general repair procedure.
Examples
Uses a time-coupled order-expiration test, an asynchronous user insert, and a flickering-warning analogy.
What you can do after
Diagnose likely timing or synchronization causes and question whether retries conceal symptoms instead of removing causes.

Read how Build High Value Software Systems explains flaky test →

100 Java Mistakes and How to Avoid Them

A flaky test is an automated test whose result varies between runs despite unchanged code and inputs.

How it is used
Uses flaky tests to explain how unstable external state undermines continuous integration.
How deep it goes
Explains timing and filesystem race conditions with focused Java snippets, then stops at practical avoidance guidance.
Examples
Uses a backup-folder timestamp assertion and filesystem file-size checks to demonstrate changing state between operations.
What you can do after
Diagnose timing and state-change failures, then stabilize tests by reusing observations and reducing separate filesystem queries.

Read how 100 Java Mistakes and How to Avoid Them explains flaky test →

Also covered in

What to know first

Related concepts

Questions about flaky test

What is flaky test?

A flaky test is a test whose outcome varies across repeated executions under equivalent conditions without a relevant change to the software or test intent.

Which Manning books explain flaky test?

The Art of Unit Testing 3e, Vibe Engineering, Build High Value Software Systems and 100 Java Mistakes and How to Avoid Them.

What should I know before learning flaky test?

Manning books that teach flaky test usually introduce integration test, software testing, assertion and bug first.

What is the difference between flaky test and trust?

A flaky test is a test whose outcome varies across repeated executions under equivalent conditions without a relevant change to the software or test intent. Trust is confidence that an entity or source is reliable and appropriate to rely on in a given context.

What is flaky test a kind of?

Flaky test is a kind of bug. A software bug is a defect that can cause software to behave incorrectly or differently from its intended or specified behavior.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.