liveBook · Manning

Flaky test

As explained in Vibe Engineering

A flaky test is an automated test that intermittently passes or fails without a corresponding change in the behavior being tested.

A flaky test intermittently passes or fails without a corresponding change in the behavior under test. It makes continuous integration (CI) feedback unreliable: a failure may reflect noise rather than a defect in the change. section 10.4

This is especially costly for coding agents. An agent may interpret a random failure as evidence, make an unnecessary fix, and treat a subsequent pass as confirmation. That false causal chain can contaminate later decisions. Faster CI does not solve an unreliable test signal; fix the lies before optimizing speed. section 10.4 section 10.9

Make flakiness visible, then reduce its effect on the feedback loop:

  • Track each test’s rerun count over 30 days and the percentage of reruns that change its result. Use the dashboard to identify unreliable tests and prioritize fixes. section 10.4.2
  • Quarantine high-impact tests into a separate diagnostic suite. Running them nightly preserves diagnostic information while excluding them from the agent’s CI feedback loop; the tradeoff is that they no longer gate that loop. section 10.4.2
  • Fix underlying causes incrementally, such as timing dependencies or mocks racing with the event loop, rather than treating retries as a permanent solution. section 10.4.2

A small set of costly tests

In a 2,400-test monorepo, about 300 tests were flaky. Twelve tests with rerun rates above 40% accounted for roughly 60% of the team’s wasted agent iterations. After those tests were quarantined and other causes addressed, agent iteration waste fell by roughly 40% over six weeks. section 10.4 section 10.4.2

Reliability is not the same as correctness. A consistently passing circular test can still repeat the implementation’s mistake. Use human-written or ticket-derived acceptance tests to check business behavior against requirements, independently of the implementation. section 10.4.1 section 10.4.2

How Vibe Engineering covers it

How deep it goes
Details rerun tracking, quarantine, and incremental repair, with timing dependencies and mock races as examples.
Examples
Uses a 2,400-test monorepo and twelve high-rerun tests to illustrate agent iteration waste and quarantine.
What you can do after
Diagnose unreliable CI feedback and prioritize quarantine or repairs using rerun data.
How it connects
Connects flaky tests to continuous integration, coding agents, circular tests, and acceptance tests.

Also explained in

Compare how 4 books explain flaky test →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.