Shadow testing, also called shadow traffic or dark loading, sends copies of live requests to a candidate system while the current production system continues to serve users. Candidate outputs are recorded for analysis, not returned to users. This lets teams evaluate technical readiness under realistic traffic without exposing users to experimental responses. section 4.3 section 8.4.2
A typical shadow test follows this path:
- A router, API gateway, or backend service duplicates or sanitizes an incoming request without changing its normal flow. section 4.3.1
- The original goes to production; the copy goes to the candidate system, preferably asynchronously. Only the production response reaches the user. section 4.3.1
- Log both systems’ responses and performance for comparison. Track latency distributions, throughput, errors, resource use, fallback behavior, and output differences. section 4.3 section 4.3.1
Figure 4.1 Illustrates how shadow traffic works where the current system design is invoked (black dotted box) and the new ranking backend service that invokes a v2 version of the model that is up evaluation being tested in the shadow system (blue dotted box).
Follow the production request path alongside the candidate ranking backend’s shadow path. section 4.3.1
Real traffic can reveal unusual request shapes, spikes, dependency failures, and expensive feature retrieval that synthetic tests miss. Evaluate the whole request pipeline, not just model inference. Set performance thresholds before using the results to decide whether the candidate is technically ready. section 4.3 section 4.3.3 section 4.3.4
Keep the shadow write path separate: candidate outputs should not update user state, caches, or feedback stores. Also plan for extra load on shared infrastructure, such as feature databases, metadata services, and caches. Users do not see candidate responses, but shared-resource load can still affect the surrounding system. section 8.4.2 section 4.3.3 section 4.3.2
Recommendation model latency
A simplified simulation routes 500 requests to current and candidate recommendation models. The models simulate average latencies of 150 ms and 220 ms, respectively. Compare latency histograms and percentile curves against predefined thresholds; the exercise focuses on system performance rather than recommendation logic. section 4.3.4
Probe edge cases
For search, replay rare, multilingual, or adversarial queries through the shadow model and score outputs for relevance, language coverage, or safety. Tag requests by edge-case class, log model version and timing, and track per-label pass rates. Use explicit robustness criteria to help gate promotion. section 8.4.2
Shadow testing cannot measure whether users prefer a candidate because they never see its output. Use it to address technical risks first, then run an A/B test to measure user, product, and business impact. section 4.3 section 4.3.2


