liveBook · Manning

Test oracle

As explained in Vibe Engineering

A test oracle is a mechanism or specification that determines whether a program's output or behavior is correct for a test input.

A test oracle determines whether a program's output or behavior is correct for a test input. In agent-assisted development, correctness requires a specification-level check, not merely evidence that generated code behaves as intended. section 10.4.2

Agent-generated unit tests can detect regressions against the agent's intent, but they cannot establish that the implementation satisfies product requirements. Human-written or ticket-derived acceptance tests provide that independent oracle. section 10.4.2

Use test oracles to separate genuine correctness signals from misleading test activity:

  • Require at least one human-written or ticket-derived acceptance test for agent-generated changes to business logic. section 10.4.2
  • Track oracle reliability with a Test Oracle Reliability Score: the percentage of test failures that represent real regressions rather than flakes. section 10.4.2
  • Treat flaky failures as noise requiring visibility and quarantine, not as evidence that the implementation is incorrect. section 10.4.2

Currency rounding regression

An agent-generated test may confirm that code matches the agent's intended behavior, while a human acceptance test catches a currency-rounding violation against the product specification. section 10.4.2

How Vibe Engineering covers it

How deep it goes
Distinguishes regression detection from correctness verification, but stops short of defining how to construct an oracle.
Examples
Uses agent-generated unit tests, human acceptance tests, currency rounding, and a Test Oracle Reliability Score.
What you can do after
Evaluate implementation correctness by adding independent acceptance tests and distinguishing regressions from flaky failures.
How it connects
Connects acceptance testing with regression detection, flaky tests, continuous integration, and specification-level verification.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.