liveBook · Manning

Shadow testing

As explained in AI Model Evaluation

A technique that sends copies of live production requests to a non-serving candidate system and records its outputs without affecting user-visible behavior.

Shadow testing, also called shadow traffic or dark loading, sends copies of live requests to a candidate system while the current production system continues to serve users. Candidate outputs are recorded for analysis, not returned to users. This lets teams evaluate technical readiness under realistic traffic without exposing users to experimental responses. section 4.3 section 8.4.2

A typical shadow test follows this path:

  • A router, API gateway, or backend service duplicates or sanitizes an incoming request without changing its normal flow. section 4.3.1
  • The original goes to production; the copy goes to the candidate system, preferably asynchronously. Only the production response reaches the user. section 4.3.1
  • Log both systems’ responses and performance for comparison. Track latency distributions, throughput, errors, resource use, fallback behavior, and output differences. section 4.3 section 4.3.1
Figure 4.1 Illustrates how shadow traffic works where the current system design is invoked (black dotted box) and the new ranking backend service that invokes a v2 version of the model that is up evaluation being tested in the shadow system (blue dotted box).

Follow the production request path alongside the candidate ranking backend’s shadow path. section 4.3.1

Real traffic can reveal unusual request shapes, spikes, dependency failures, and expensive feature retrieval that synthetic tests miss. Evaluate the whole request pipeline, not just model inference. Set performance thresholds before using the results to decide whether the candidate is technically ready. section 4.3 section 4.3.3 section 4.3.4

Keep the shadow write path separate: candidate outputs should not update user state, caches, or feedback stores. Also plan for extra load on shared infrastructure, such as feature databases, metadata services, and caches. Users do not see candidate responses, but shared-resource load can still affect the surrounding system. section 8.4.2 section 4.3.3 section 4.3.2

Recommendation model latency

A simplified simulation routes 500 requests to current and candidate recommendation models. The models simulate average latencies of 150 ms and 220 ms, respectively. Compare latency histograms and percentile curves against predefined thresholds; the exercise focuses on system performance rather than recommendation logic. section 4.3.4

Probe edge cases

For search, replay rare, multilingual, or adversarial queries through the shadow model and score outputs for relevance, language coverage, or safety. Tag requests by edge-case class, log model version and timing, and track per-label pass rates. Use explicit robustness criteria to help gate promotion. section 8.4.2

Shadow testing cannot measure whether users prefer a candidate because they never see its output. Use it to address technical risks first, then run an A/B test to measure user, product, and business impact. section 4.3 section 4.3.2

How AI Model Evaluation covers it

How deep it goes
Details routing, logging, isolation, metrics, and promotion criteria while simplifying recommendation logic in the simulation.
Examples
Simulates 500 recommendation requests and probes rare or multilingual search queries with rubric scoring.
What you can do after
Evaluate technical readiness and gate promotion using performance thresholds and robustness criteria.
How it connects
Connects production readiness with A/B testing, offline evaluation, and robustness evaluation.

Also explained in

Compare how 3 books explain shadow testing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.