An A/B test is an online controlled experiment that measures a change by comparing outcomes for users assigned to a test variant with outcomes for a control group. For an AI model, it reveals what happens when predictions interact with real users, product surfaces, and latency constraints—not just historical data. section 6.1 section 6.0
Offline evaluations provide faster, repeatable evidence for selecting and refining candidates, but they cannot replace measurement of real user impact. Shadow traffic can establish technical readiness without showing users the candidate output; an A/B test then measures its effect on users and the business. section 1.6 section 4.3.2
Design the experiment from its purpose outward:
- Define the user or product behavior the change should influence, and state a hypothesis before looking at results. section 6.3 section 6.6
- Choose a control and treatment, assignment logic, primary success metrics, and guardrail metrics. An existing production model or a heuristic can serve as the control. section 6.3 section 1.6 section 6.3
- Plan sample size, duration, and rollout timing. More variants increase complexity and can require larger samples or longer tests; reaching enough power may take weeks. section 6.8 section 1.6
Figure 6.2 A/B testing design layers, organized from outer to inner circles. The closer to the core, the more foundational and irreversible the decisions tend to be: the outer execution layer covers operational factors like test duration, rollout timing, and sample size; the middle experimental design layer defines how the test is structured, including variants, metrics, and assignment logic; and the inner core layer captures the underlying purpose of the model and the user or product behavior it’s intended to influence.
The design layers connect the model’s purpose to experimental structure and execution choices. section 6.3
Testing a movie recommender
A streaming platform compares its existing recommendation model with a version that adds device type, time of day, and recent searches. The hypothesis is that contextual features improve relevance, click-through rate, and watch time without increasing latency or reducing satisfaction. The decision also depends on whether important user cohorts receive worse recommendations and whether the gain justifies changing a working system. section 6.3.6
Match test timing to risk and learning goals. A lightweight early test can provide directional feedback, while stronger offline evidence helps identify failure modes and affected cohorts before exposure. With weaker evidence, use tighter monitoring, alerts, fallbacks, and a rollback plan. section 6.2.2 section 6.2.3
Interpret weak or conflicting results carefully. A small effect may not justify added latency, cost, or operational burden; a poorly aligned metric or underpowered test may obscure a useful change. For offline–online mismatches, compare segments, inspect logs, validate inputs and exposure, and revisit the offline evaluation. Cold starts, feature drift, and novelty effects can also distort model experiments. section 6.6 section 6.5.5 section 6.8




