liveBook · Manning

Shadow evaluation

As explained in Vibe Engineering

Shadow evaluation is a technique that runs a reference evaluation alongside a production evaluation and compares their results without replacing the production path.

Shadow evaluation runs a reference evaluation alongside a production path, compares their results, and leaves the production decision unchanged. It turns an assumption about a cheaper or optimized model into measured evidence rather than accepted risk. section 9.2.4 section 9.4.3

Use the pattern as a controlled quality check:

  • Sample production cases randomly instead of duplicating every evaluation. section 9.2.4
  • Run each sampled case through both the optimized path and the reference path. section 9.4.3
  • Log disagreements and review them regularly to identify systematic false negatives. section 9.2.4 section 9.4.3
  • Reduce the sample only after stable measurements justify lower monitoring cost. section 9.4.4

CI review pipeline

A CI pipeline reviews one in twenty pull requests through both a Haiku-first optimized workflow and a Sonnet-only reference workflow. The team logs disagreements, reviews them weekly, and spends about $30 monthly on the additional calls. After three stable months, it reduces the shadow rate from 5% to 2%. section 9.4.3 section 9.4.4

How Vibe Engineering covers it

How deep it goes
Defines parallel evaluation, sampling, disagreement logging, and review, but stops short of specifying statistical confidence methods.
Examples
Uses Haiku and Sonnet reviews, a one-in-twenty sample, logged disagreements, and a 5%-to-2% monitoring reduction.
What it assumes
Assumes an optimized evaluation path, a trusted reference path, comparable outputs, and a process for reviewing disagreements.
What you can do after
Diagnose quality drift and quantify false negatives before trusting a cheaper evaluation path.
How it connects
Connects model routing with CI/CD pipelines, reference models, quality monitoring, and false-negative analysis.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.