A test oracle determines whether a program's output or behavior is correct for a test input. In agent-assisted development, correctness requires a specification-level check, not merely evidence that generated code behaves as intended. section 10.4.2
Agent-generated unit tests can detect regressions against the agent's intent, but they cannot establish that the implementation satisfies product requirements. Human-written or ticket-derived acceptance tests provide that independent oracle. section 10.4.2
Use test oracles to separate genuine correctness signals from misleading test activity:
- Require at least one human-written or ticket-derived acceptance test for agent-generated changes to business logic. section 10.4.2
- Track oracle reliability with a Test Oracle Reliability Score: the percentage of test failures that represent real regressions rather than flakes. section 10.4.2
- Treat flaky failures as noise requiring visibility and quarantine, not as evidence that the implementation is incorrect. section 10.4.2
Currency rounding regression
An agent-generated test may confirm that code matches the agent's intended behavior, while a human acceptance test catches a currency-rounding violation against the product specification. section 10.4.2
