liveBook · Manning

Flaky test

As explained in Build High Value Software Systems

A test whose result is nondeterministic despite unchanged software and test inputs.

A flaky test produces different results across identical executions when the code and environment have not changed. It weakens build feedback: developers may rerun builds, batch changes, or learn to ignore failures. A red build then stops reliably meaning “stop and investigate.” section 5.4 section 5.4.1

Look for both the source of variability and the costs it spreads through the workflow:

  • Timing assumptions, such as system time or background work completing later than expected, can make results intermittent. section 5.4.1
  • Network latency or shared test data can also make outcomes vary without a code change. section 5.4
  • Reruns consume CI time; retries, sleeps, and ignored failures can mask defects and make unreliable signals part of normal workflow. section 5.4.1

A time-coupled expiration test

A test creates an order, advances time by 30 minutes, processes expirations, and checks that the order expired. If a background worker runs late under load, the assertion may fail in CI but pass on a rerun. The changing timing—not a changed assertion—makes the result unreliable. section 5.4.1

Listing 5.3 A time-coupled expiration test
def test_order_expires_after_30_minutes():
    order = create_order()
 
    advance_time(minutes=30)
    process_expirations()
 
    assert order.status == "expired"

The listing shows how the test couples its assertion to a timed expiration workflow. section 5.4.1

Retrying can hide the cause

An intermittent user lookup may tempt a developer to add retries. If the test helper starts an asynchronous database insert without awaiting it, the lookup races the write; awaiting the insert addresses the identified cause, while retries can conceal it. section 8.4

How Build High Value Software Systems covers it

How deep it goes
Defines nondeterminism under unchanged code and environment, then traces workflow costs; stops short of a general repair procedure.
Examples
Uses a time-coupled order-expiration test, an asynchronous user insert, and a flickering-warning analogy.
What you can do after
Diagnose likely timing or synchronization causes and question whether retries conceal symptoms instead of removing causes.
How it connects
Links flaky tests to CI, timing dependencies, shared test state, cycle time, and root-cause analysis.

Also explained in

Compare how 4 books explain flaky test →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build High Value Software Systems. The section links point to the parts of the book it draws on.