Shadow testing
Shadow testing evaluates a candidate system by sending it copies of live requests while preventing its outputs from affecting user-facing behavior.
AI Model Evaluation
A technique that sends copies of live production requests to a non-serving candidate system and records its outputs without affecting user-visible behavior.
- How it is used
- Tests candidate AI systems on realistic production traffic before exposing users to their outputs.
- How deep it goes
- Details routing, logging, isolation, metrics, and promotion criteria while simplifying recommendation logic in the simulation.
- Examples
- Simulates 500 recommendation requests and probes rare or multilingual search queries with rubric scoring.
- What you can do after
- Evaluate technical readiness and gate promotion using performance thresholds and robustness criteria.
The Coder Cafe
Shadow testing runs a change in parallel with production traffic while preventing it from affecting real users.
- How it is used
- Reduces deployment risk by validating changes against production traffic before exposing them to real users.
- How deep it goes
- Defines parallel execution and safe validation but stops short of implementation details or operational procedures.
- Examples
- Uses a proposed change processing production traffic in parallel without affecting real users.
- What you can do after
- Evaluate a change’s correctness before enabling it for real users.
Building Reliable AI Systems
Shadow testing evaluates a system or change alongside a live system without allowing the test results to affect user-facing behavior.
- How it is used
- Uses live queries and traffic patterns to evaluate candidate models before deployment without risking user experience.
- How deep it goes
- Implements asynchronous candidate requests and response logging, and notes cost and duration trade-offs.
- Examples
- Uses real user queries and an asynchronous Python sketch that logs production and candidate responses.
- What you can do after
- Evaluate candidate models on production traffic and inform deployment decisions without exposing candidate outputs to users.
Read how Building Reliable AI Systems explains shadow testing →
Also covered in
- ScyllaDB in Action
- Rearchitecting LLMsIn early access (MEAP)
- AI Agents in Action, Second Edition
- Kubernetes in Action, Second Edition
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
Related concepts
- A/B testcontrasts with · An A/B test is a controlled experiment that compares outcomes for two variants assigned to different users or experimental units.
- benchmarkcontrasts with · A benchmark is a defined, repeatable task, workload, test, dataset, or reference standard used to measure or compare systems or methods under specified conditions.
- small language modeluses · A small language model is a comparatively compact language model with limited parameters or capacity and typically lower computational and storage requirements than larger models.
- infrastructureuses · Infrastructure is the foundational physical, digital, and operational resources and systems that support the operation of organizations, computing systems, or services.
- ingress objectuses · A Kubernetes Ingress is an API resource that specifies rules for routing external HTTP and HTTPS requests to services within a cluster.
Questions about shadow testing
What is shadow testing?
Shadow testing evaluates a candidate system by sending it copies of live requests while preventing its outputs from affecting user-facing behavior.
Which Manning books explain shadow testing?
AI Model Evaluation, The Coder Cafe and Building Reliable AI Systems.
What should I know before learning shadow testing?
Manning books that teach shadow testing usually introduce language model first.
What is the difference between shadow testing and A/B test?
Shadow testing evaluates a candidate system by sending it copies of live requests while preventing its outputs from affecting user-facing behavior. An A/B test is a controlled experiment that compares outcomes for two variants assigned to different users or experimental units.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


