liveBook · Manning

Test oracle reliability score

As explained in Vibe Engineering

Test oracle reliability score is the percentage of test outcomes that correctly indicate whether the tested behavior is acceptable.

Fast CI can still send an agent toward the wrong fix if test failures are flaky. The test oracle reliability score (TORS) measures the percentage of test failures that are real regressions rather than flakes. A low score means more investigation is spent on noise, so reliability should be addressed before CI speed. section 10.4.2

A reliable failure signal is not the same as a complete correctness check. Agent-generated unit tests can detect regressions against what the agent intended, while human-written or ticket-derived acceptance tests check whether that intent matches product requirements. section 10.4.2

Use TORS as a preventive CI governance metric:

  • Track it per module and review it weekly. A target above 95% provides an early warning when reliability falls. section 10.7.1
  • Make flakiness visible by tracking reruns per test and the percentage of reruns that change the result. Separate and quarantine flaky tests rather than letting them misdirect iterations. section 10.4.2 section 10.7.1

A module drops below target

After a test refactoring sprint, a notification module’s TORS fell to 88%. Its dashboard flagged the drop; the team quarantined three tests, and the score returned to 96%. section 10.7.1

How Vibe Engineering covers it

How deep it goes
Defines the failure-based percentage and gives a monitoring and quarantine workflow, without specifying a calculation procedure.
Examples
Uses a notification module’s fall from 88% to 96% after three tests were quarantined.
What you can do after
Evaluate test feedback reliability by tracking TORS per module and investigating flaky tests before optimizing CI speed.
How it connects
Connects test flakiness, regression testing, acceptance testing, and continuous integration feedback.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.