liveBook · Manning

Shadow testing

Shadow testing evaluates a candidate system by sending it copies of live requests while preventing its outputs from affecting user-facing behavior.

How 3 Manning books explain it

AI Model Evaluation

A technique that sends copies of live production requests to a non-serving candidate system and records its outputs without affecting user-visible behavior.

How it is used
Tests candidate AI systems on realistic production traffic before exposing users to their outputs.
How deep it goes
Details routing, logging, isolation, metrics, and promotion criteria while simplifying recommendation logic in the simulation.
Examples
Simulates 500 recommendation requests and probes rare or multilingual search queries with rubric scoring.
What you can do after
Evaluate technical readiness and gate promotion using performance thresholds and robustness criteria.

Read how AI Model Evaluation explains shadow testing →

The Coder Cafe

Shadow testing runs a change in parallel with production traffic while preventing it from affecting real users.

How it is used
Reduces deployment risk by validating changes against production traffic before exposing them to real users.
How deep it goes
Defines parallel execution and safe validation but stops short of implementation details or operational procedures.
Examples
Uses a proposed change processing production traffic in parallel without affecting real users.
What you can do after
Evaluate a change’s correctness before enabling it for real users.

Read how The Coder Cafe explains shadow testing →

Building Reliable AI Systems

Shadow testing evaluates a system or change alongside a live system without allowing the test results to affect user-facing behavior.

How it is used
Uses live queries and traffic patterns to evaluate candidate models before deployment without risking user experience.
How deep it goes
Implements asynchronous candidate requests and response logging, and notes cost and duration trade-offs.
Examples
Uses real user queries and an asynchronous Python sketch that logs production and candidate responses.
What you can do after
Evaluate candidate models on production traffic and inform deployment decisions without exposing candidate outputs to users.

Read how Building Reliable AI Systems explains shadow testing →

Also covered in

What to know first

Related concepts

Questions about shadow testing

What is shadow testing?

Shadow testing evaluates a candidate system by sending it copies of live requests while preventing its outputs from affecting user-facing behavior.

Which Manning books explain shadow testing?

AI Model Evaluation, The Coder Cafe and Building Reliable AI Systems.

What should I know before learning shadow testing?

Manning books that teach shadow testing usually introduce language model first.

What is the difference between shadow testing and A/B test?

Shadow testing evaluates a candidate system by sending it copies of live requests while preventing its outputs from affecting user-facing behavior. An A/B test is a controlled experiment that compares outcomes for two variants assigned to different users or experimental units.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.