A flaky test intermittently passes or fails without a corresponding change in the behavior under test. It makes continuous integration (CI) feedback unreliable: a failure may reflect noise rather than a defect in the change. section 10.4
This is especially costly for coding agents. An agent may interpret a random failure as evidence, make an unnecessary fix, and treat a subsequent pass as confirmation. That false causal chain can contaminate later decisions. Faster CI does not solve an unreliable test signal; fix the lies before optimizing speed. section 10.4 section 10.9
Make flakiness visible, then reduce its effect on the feedback loop:
- Track each test’s rerun count over 30 days and the percentage of reruns that change its result. Use the dashboard to identify unreliable tests and prioritize fixes. section 10.4.2
- Quarantine high-impact tests into a separate diagnostic suite. Running them nightly preserves diagnostic information while excluding them from the agent’s CI feedback loop; the tradeoff is that they no longer gate that loop. section 10.4.2
- Fix underlying causes incrementally, such as timing dependencies or mocks racing with the event loop, rather than treating retries as a permanent solution. section 10.4.2
A small set of costly tests
In a 2,400-test monorepo, about 300 tests were flaky. Twelve tests with rerun rates above 40% accounted for roughly 60% of the team’s wasted agent iterations. After those tests were quarantined and other causes addressed, agent iteration waste fell by roughly 40% over six weeks. section 10.4 section 10.4.2
Reliability is not the same as correctness. A consistently passing circular test can still repeat the implementation’s mistake. Use human-written or ticket-derived acceptance tests to check business behavior against requirements, independently of the implementation. section 10.4.1 section 10.4.2



