liveBook · Manning

A/B test

As explained in AI Model Evaluation

A controlled online experiment that compares outcomes between two variants, typically a control and a treatment, using assigned users or traffic.

An A/B test is an online controlled experiment that measures a change by comparing outcomes for users assigned to a test variant with outcomes for a control group. For an AI model, it reveals what happens when predictions interact with real users, product surfaces, and latency constraints—not just historical data. section 6.1 section 6.0

Offline evaluations provide faster, repeatable evidence for selecting and refining candidates, but they cannot replace measurement of real user impact. Shadow traffic can establish technical readiness without showing users the candidate output; an A/B test then measures its effect on users and the business. section 1.6 section 4.3.2

Design the experiment from its purpose outward:

  • Define the user or product behavior the change should influence, and state a hypothesis before looking at results. section 6.3 section 6.6
  • Choose a control and treatment, assignment logic, primary success metrics, and guardrail metrics. An existing production model or a heuristic can serve as the control. section 6.3 section 1.6 section 6.3
  • Plan sample size, duration, and rollout timing. More variants increase complexity and can require larger samples or longer tests; reaching enough power may take weeks. section 6.8 section 1.6
Figure 6.2 A/B testing design layers, organized from outer to inner circles. The closer to the core, the more foundational and irreversible the decisions tend to be: the outer execution layer covers operational factors like test duration, rollout timing, and sample size; the middle experimental design layer defines how the test is structured, including variants, metrics, and assignment logic; and the inner core layer captures the underlying purpose of the model and the user or product behavior it’s intended to influence.

The design layers connect the model’s purpose to experimental structure and execution choices. section 6.3

Testing a movie recommender

A streaming platform compares its existing recommendation model with a version that adds device type, time of day, and recent searches. The hypothesis is that contextual features improve relevance, click-through rate, and watch time without increasing latency or reducing satisfaction. The decision also depends on whether important user cohorts receive worse recommendations and whether the gain justifies changing a working system. section 6.3.6

Match test timing to risk and learning goals. A lightweight early test can provide directional feedback, while stronger offline evidence helps identify failure modes and affected cohorts before exposure. With weaker evidence, use tighter monitoring, alerts, fallbacks, and a rollback plan. section 6.2.2 section 6.2.3

Interpret weak or conflicting results carefully. A small effect may not justify added latency, cost, or operational burden; a poorly aligned metric or underpowered test may obscure a useful change. For offline–online mismatches, compare segments, inspect logs, validate inputs and exposure, and revisit the offline evaluation. Cold starts, feature drift, and novelty effects can also distort model experiments. section 6.6 section 6.5.5 section 6.8

How AI Model Evaluation covers it

How deep it goes
Develops layered experiment design, readiness, model-specific risks, and interpretation of ambiguous results.
Examples
Uses a contextual movie recommender test and a latency degradation experiment with control and treatment variants.
What it assumes
Assumes a functioning A/B testing platform when discussing prerequisites for AI model experiments.
What you can do after
Evaluate a model change by designing an online experiment and interpreting its results against product goals and operational trade-offs.
How it connects
Connects online controlled experiments with offline evaluation, shadow traffic, statistical power, and feature drift.

Also explained in

Compare how 5 books explain A/B test →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.