Fast CI can still send an agent toward the wrong fix if test failures are flaky. The test oracle reliability score (TORS) measures the percentage of test failures that are real regressions rather than flakes. A low score means more investigation is spent on noise, so reliability should be addressed before CI speed. section 10.4.2
A reliable failure signal is not the same as a complete correctness check. Agent-generated unit tests can detect regressions against what the agent intended, while human-written or ticket-derived acceptance tests check whether that intent matches product requirements. section 10.4.2
Use TORS as a preventive CI governance metric:
- Track it per module and review it weekly. A target above 95% provides an early warning when reliability falls. section 10.7.1
- Make flakiness visible by tracking reruns per test and the percentage of reruns that change the result. Separate and quarantine flaky tests rather than letting them misdirect iterations. section 10.4.2 section 10.7.1
A module drops below target
After a test refactoring sprint, a notification module’s TORS fell to 88%. Its dashboard flagged the drop; the team quarantined three tests, and the score returned to 96%. section 10.7.1
