A green pipeline can hide a circular test: a test derived from an implementation rather than an independent requirement. The test and code may share the same misunderstanding, so their agreement gives false confidence instead of verifying the required behavior. This risk is especially easy to miss when an agent writes both the code and its tests. section 10.4.1
A rounding error that passes
A discount calculation truncates the result for a 50% discount on 9.99 to 4.99. A test derived from that implementation asserts 4.99 and passes, although the required rounded result is 5.00. A specification-derived acceptance test would check the required result instead. section 10.4.1
Replace agreement with independent checks:
- Write acceptance tests from product requirements, or generate them from ticket requirements rather than from the implementation. Treat agent-written code and tests as unverified until an independent check confirms the behavior. section 10.4.1
- Use mutation testing to probe the suite: change a comparison, delete a condition, or change a return value, then see whether tests catch it. A surviving mutation is a warning to inspect the affected assertions against the specification. section 10.4.2
- Judge test quality alongside coverage. Rewriting checkout tests against the specification reduced coverage from 94% to 87%, while mutation survival fell from 23% to 6%. section 10.4.2
