Shadow evaluation runs a reference evaluation alongside a production path, compares their results, and leaves the production decision unchanged. It turns an assumption about a cheaper or optimized model into measured evidence rather than accepted risk. section 9.2.4 section 9.4.3
Use the pattern as a controlled quality check:
- Sample production cases randomly instead of duplicating every evaluation. section 9.2.4
- Run each sampled case through both the optimized path and the reference path. section 9.4.3
- Log disagreements and review them regularly to identify systematic false negatives. section 9.2.4 section 9.4.3
- Reduce the sample only after stable measurements justify lower monitoring cost. section 9.4.4
CI review pipeline
A CI pipeline reviews one in twenty pull requests through both a Haiku-first optimized workflow and a Sonnet-only reference workflow. The team logs disagreements, reviews them weekly, and spends about $30 monthly on the additional calls. After three stable months, it reduces the shadow rate from 5% to 2%. section 9.4.3 section 9.4.4
