liveBook · Manning

Chapter 6: Evaluating models in an A/B test

What you will learn in chapter 6 of AI Model Evaluation

The chapter introduction explains why offline evaluations are necessary but A/B tests are needed to judge a model’s effect on real users and product outcomes.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduction explains why offline evaluations are necessary but A/B tests are needed to judge a model’s effect on real users and product outcomes.

Key terms: offline evaluations, online data, A/B test, online controlled experiments, offline metrics, user metrics, local metric, downstream experience, user behavior

6.1 A/B testing

The section introduces A/B testing as a way to measure a model’s impact with real users, beyond what offline evaluations can predict.

Key terms: offline evaluations, directional confidence, online controlled experiment, A/B test, A/B testing, test group, control group, treatment, variant, key metrics

6.1.1 Why A/B test

This section explains why A/B tests are needed to measure a model’s real product impact rather than relying on offline predictions.

Key terms: offline evaluation, A/B test, online, real user behavior, product experience, real product impact, A/B testing is a mindset, product better

6.2 The ‘right time’ to A/B test a model

This section frames A/B-test readiness as a decision based on model maturity, timelines, and offline evidence rather than a guarantee of online success.

Key terms: model maturity, online evaluation, offline evaluation practices, offline exit criteria, baseline, primary offline metric, guardrail metrics, segment-level diagnostics, error analysis, offline evaluations

6.2.1 Model maturity

Model maturity determines whether an A/B test should explore an early idea’s viability or precisely evaluate a new version of an established model.

Key terms: early-stage models, offline evaluation suite, feature-level metrics, model maturity, top-line metrics, feature level metrics, regressions

6.2.2 Timelines

This section explains how to time model A/B tests to get useful feedback without assuming every model needs the same testing schedule.

Key terms: lightweight, iterative A/B tests, minimum viable product (MVP)-style approach, directional signals, risk tolerance, offline signals, fail fast, sensitive, fast-moving metrics

6.2.3 How offline evaluations influence A/B test readiness

This section explains how the strength of offline evaluations changes the risk, timing, and safeguards of a model A/B test.

Key terms: offline evaluations, directional confidence, failure modes, segment-level behavior, higher-risk learning exercise, observability, real-time dashboards, alerts, fallbacks, model maturity

6.3 Designing an A/B test

Introduces model A/B test design as a comparison against an appropriate control, then frames design decisions as layers from logistics to core product value.

Key terms: A/B testing, control group, heuristic, model version, layers, surface-level logistics, core aspects

6.3.1 Operational setup

This section identifies the execution decisions for an A/B test and explains how exposure rules determine where and with whom to run it.

Key terms: sample size, duration, traffic allocation, exposure rules, ramp up, targeting criteria, device type, geographic region, user tenure

6.3.2 Experimental design

This section explains how an A/B test’s experimental design defines what changes, who receives each variant, and how model impact is measured and interpreted.

Key terms: experimental design, unit of randomization, segmentation plans, operational rollout, test variant, control variant, targeting criteria, success metrics, guardrail metrics, user retention

6.3.3 Hypothesis and purpose

The section explains how an A/B test’s purpose shapes its hypothesis, success criteria, and decisions about rollout or further iteration.

Key terms: hypothesis, success criteria, decision framework, exposure rules

6.3.4 Model purpose & intent

This section explains why an A/B test should be grounded in the model’s intended role and product impact, not just a numerical result.

Key terms: model purpose & intent, intended user behavior, product role of the model, risk profile and acceptable trade-offs, assumptions, hypotheses, metrics, ambiguous results, false wins, product goals

6.3.5 Experiment validity checks before interpreting results

This section explains the checks needed to decide whether an A/B test’s results can be trusted as evidence of a model’s effect.

Key terms: assignment, exposure, treatment variant, exposure logging, randomization, unit of randomization, sample ratio, pre-planned segments, stopping rules, model version logging

6.3.6 Movie recommendation A/B test example

The section applies an A/B test design to a proposed movie recommender update, from its product hypothesis and variants through its metrics, operational setup, and launch decisions.

Key terms: contextual features, viewing intent, user embeddings, content embeddings, primary metrics, secondary metric, guardrail metrics, launch criterion, control, Treatment A

6.4 Prerequisites for running A/B tests

This introduction explains why A/B tests of AI models need preparation beyond that of traditional product experiments, without yet detailing the prerequisites.

Key terms: model behavior, inference cost, infra issues

6.4.1 Latency monitoring

This section explains how inference latency affects users and can confound A/B test results, making real-time latency monitoring an experiment guardrail.

Key terms: inference latency, treatment model, control, confound, guardrail metric, real time, model version, treatment group

6.4.2 Feature drift monitoring

This section explains how feature drift can undermine a model A/B test and how to detect and respond to significant shifts in production inputs.

Key terms: feature drift, input features, training data, production data, Kolmogorov-Smirnov Test, KS test, Population Stability Index, PSI, drift monitoring, key features

6.4.3 Logging still matters

This section explains why model A/B tests need logs that connect experiment assignment to what users actually saw and did, so results can be interpreted reliably.

Key terms: experiment assignment, exposure events, model version, model output, downstream user outcomes, serving path, fallback path, cache, safety layer, latency

6.5 Quirks of A/B testing models

This section explains why AI model A/B tests need more careful interpretation than typical UX experiments.

Key terms: cold start effects, high variances, training data bias, A/B test

6.5.1 Multiple model variants increase complexity

This section explains why testing multiple model variants demands more traffic, creates false-positive risk, and requires a clear hypothesis for each variant.

Key terms: treatment variant, sample size, exposure, multiple-comparisons problem, false positive, Bonferroni correction, false discovery rate (FDR) correction, hypothesis, test variant, interactions

6.5.2 Novelty effect

Explains how initial user curiosity can inflate A/B test results and how to check whether an apparent lift lasts.

Key terms: novelty effect, treatment metrics, early lift, primary metric, week-over-week, experiment window

6.5.3 Model warm-up and cold start effects

This section explains how model warm-up and cold-start effects can distort early A/B test results and how to plan when primary analysis should begin.

Key terms: warm-up, cold-start behavior, treatment, primary analysis, minimum stabilization period, warm-up period, interaction volume, stability-based

6.5.4 Variance and sensitivity to noise

Explains why noise can make AI A/B test results unstable and how experiment design can improve the ability to detect a model’s effect.

Key terms: high variance, observed effect sizes, control experience, false positives or negatives, sample size, stable metrics, robust evaluation windows, small effect sizes, long feedback loops, rare conversions

6.5.5 Offline and online mismatch

This section explains why strong offline results can fail to translate into A/B test gains and how to investigate the disagreement.

Key terms: offline metric, real product objective, A/B test, stale, training data, features, production, diagnostic starting point, segments, logs

6.6 Interpreting ambiguous or low-signal results

The section explains how to interpret ambiguous AI-model A/B test results without overstating the evidence and how to use them to make a launch, iteration, or stop decision.

Key terms: low-signal result, underpowered, random variation, model effect, operational risk, infrastructure cost, model’s purpose, success criteria, short-term click-through metric, segment analysis

6.7 Engineering considerations

This section identifies infrastructure and an experimentation platform as the two key engineering considerations for evaluating a model in an online A/B test.

Key terms: infrastructure reliability, latency, evaluation strategy, experimentation platform, control, test variant, offline evaluations

6.7.1 Infrastructure-induced latency or errors

Infrastructure performance belongs in model A/B test evaluation because a model can improve a success metric while harming users, system health, or cost.

Key terms: latency, instability, scaling issues, engagement, success metrics, guardrail metrics, inference latency, system throughput, CPU and GPU utilization, cost per request

6.7.2 If you just don't have an A/B testing platform

This section explains how to start controlled online model testing without a dedicated A/B testing platform, using lightweight infrastructure or a hosted service while accounting for missing guardrails.

Key terms: feature flag system, randomize exposure, user or session ID, controlled way, experimentation-as-a-service platform, randomization, assignment, exposure logging, metrics pipelines, guardrails

6.8 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 5: Counterfactual evaluations Chapter 7: From offline evaluation to live experiment →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.