Chapter 6: Evaluating models in an A/B test
The chapter introduction explains why offline evaluations are necessary but A/B tests are needed to judge a model’s effect on real users and product outcomes.
After this chapter you will be able to
- Offline and online evaluation roles. Explain why offline evaluations are needed before launch but cannot establish a model’s real impact on users, and what an A/B test adds.
- Model scores and product outcomes. Describe why an offline improvement or a better local metric may not translate into a better product experience, and why a modest offline gain can still matter online.
- Offline and online evidence. Explain why an A/B test is needed even after offline evaluation: offline results provide directional confidence, while an online experiment measures what happens when the model interacts with real users in the product.
- Test and control comparison. Describe an A/B test as measuring a change’s impact by comparing key metrics for a test group that receives the change with a control group that does not. State that the metrics should reflect meaningful user outcomes.
- Offline predictions and online evidence. Distinguish what offline evaluations predict from what an A/B test measures, and explain how results from a subset of users inform a possible launch to all users.
- Model impact within the product. Explain why offline results can differ from A/B test results and describe the kinds of product impact the test can reveal.
- Offline Exit Criteria. Describe the kinds of minimum offline conditions that make a model a candidate for online testing.
- Limits of Offline Readiness. Distinguish being ready to test online from having proven that a model will succeed online.
- Early-stage model testing strategy. Explain why an early-stage model should be tested for promise with real users rather than readiness for a full rollout, and describe the offline and A/B testing approach proposed here.
- Mature model testing strategy. Describe how the A/B testing strategy changes when evaluating a new version of a mature model, including the question asked and the evidence and safeguards emphasized.
- Early iterative A/B testing. Explain why a team prioritizing speed might run lightweight A/B tests early in model development, particularly when exploring a new model type or product application.
- Context-dependent testing timelines. Describe the factors that determine when and how thoroughly to A/B test a model, and how the timing affects the kind of signal a team needs.
- Offline evidence and test risk. Explain how strong offline evaluations reduce the risk of entering an A/B test by providing directional confidence about model quality, failure modes, user cohorts, and likely product impact.
- Weak evidence and online safeguards. Describe why an A/B test is not a substitute for understanding model behavior, and how to proceed more cautiously when offline evaluations are weak.
- Model variant and control. Explain what is introduced to users in a model A/B test and how the control differs for a brand-new model versus an updated one.
- A/B test execution decisions. Describe the operational decisions that specify how an A/B test runs: sample size, duration, traffic allocation, exposure rules, and whether assignment ramps up gradually or happens all at once.
- Strategic exposure targeting. Explain that exposure rules, also called targeting criteria, restrict the experiment to relevant users or contexts and can be used to choose a strategic starting audience.
- Experimental design choices. Describe the middle layer of A/B test design: the treatment and baseline, unit of randomization, metrics, segmentation plans, and guardrails. Explain how these choices connect the hypothesis to the operational rollout.
- Test and control variants. Explain what users in the test and control variants experience and identify the unit of randomization as a design choice.
- Success and guardrail metrics. Distinguish success metrics from guardrail metrics and explain why an improvement in a primary metric does not, by itself, establish a good outcome.
- Experiment purpose and decision criteria. Explain why an A/B test needs a clear learning objective, and describe how its hypothesis, success criteria, and decision framework make that objective actionable.
- Model purpose and intent. Explain what the model’s core intent should clarify before interpreting an A/B test: the desired change in user behavior, the model’s role in the product, acceptable risk and trade-offs, and the assumptions the test might validate or challenge.
- Assignment versus actual exposure. Distinguish assignment to a variant from actually seeing the changed model output, and explain why logging assignment alone can misrepresent who received the treatment.
- Randomization and analysis safeguards. Describe the checks on random assignment, the observed versus planned traffic split, and decisions made before versus after examining results.
- Serving and measurement integrity. Explain how to check what model behavior was served, whether other experiments or instrumentation affected the metrics, and why unresolved problems limit a launch decision.
- Contextual feature hypothesis. Explain why the team is testing contextual features in an existing recommender and what product benefit the change must deliver.
- Success metrics and guardrails. Describe the hypothesis, metric roles, and launch criterion for the recommender test.
- Control and treatment variants. Distinguish the control and two treatments and explain what their comparison reveals.
- Predefined launch decisions. Explain how the team would interpret wins, trade-offs, flat results, and segment gains before deciding what to do next.
- Model experiment complexity. Explain why an A/B test of an AI model can be harder to run reliably than an experiment changing a UX component, citing the risks named here.
- Latency and experiment interpretation. Explain what inference latency is and why a speed difference between treatment and control can make an A/B test misleading.
- Latency as experiment guardrail. Describe how teams should monitor latency during a model A/B test and what comparisons the monitoring should support.
- Feature drift and test validity. Explain what feature drift is and why it can make an offline-tested model behave unpredictably during an A/B test.
- Statistical feature drift detection. Describe how comparing training and production feature distributions can flag drift, including the roles of the KS test and PSI.
- Assignment to outcome logging. Describe the minimum data needed to interpret a model A/B test: variant assignment, actual exposure, model version, returned output, and subsequent user behavior.
- Serving path attribution. Explain how fallbacks, caches, and safety layers can make the experience differ from the assigned model, and why logging these paths matters when interpreting results.
- Model A/B test complexity. Explain why an AI model A/B test cannot be interpreted exactly like a typical UX feature test, and what can go wrong if it is.
- Traffic needs for multiple variants. Explain why adding model variants increases the traffic or duration needed for an A/B test.
- False positives across comparisons. Explain the multiple-comparisons problem in a multi-variant experiment and describe the corrections the section names.
- Purposeful variant selection. Justify giving every tested variant a clear hypothesis and a plan for interpreting its results.
- Novelty Driven Metric Lift. Explain how a changed experience can temporarily raise treatment metrics even when the model is not better.
- Time Based Lift Check. Describe how to use experiment duration and week-over-week primary-metric trends before deciding to launch.
- Cold-start effects on results. Explain why early A/B test outcomes may reflect model or system stabilization rather than long-term model quality, including for non-adaptive models.
- Predefined warm-up analysis criteria. Describe how to set warm-up criteria before an A/B test and use them to separate stabilization from primary analysis while preserving enough post-warm-up data for comparison.
- High variance in test outcomes. Explain what high variance means in an AI A/B test and why it makes the model’s effect difficult to interpret.
- Detectability through experiment design. Describe which conditions make meaningful differences harder to detect and which experiment-design choices the section recommends in response.
- Offline and product goal alignment. Explain why a high offline metric does not guarantee a successful A/B test: the metric or its labels may not reflect the user behavior the product actually values.
- Investigating conflicting evaluation results. Describe how to treat an offline–online disagreement as a diagnostic starting point rather than assuming one cause, using A/B test evidence to examine segments, logs, inputs, exposure, and the offline evaluation design.
- Limits of low-signal results. Distinguish an experiment that cannot detect an effect from evidence that the model has no effect. Explain how too few users, too short a test, rare outcomes, or a highly variable metric can make results inconclusive.
- Evidence-based experiment decisions. Describe how to return to the original hypothesis and decision framework when results are ambiguous. Check validity and exposure, consider primary and guardrail metrics alongside offline evaluations, and judge whether the effect matters before deciding to launch, iterate and retest, or stop.
- Infrastructure as evaluation concern. Explain why model infrastructure is part of an online A/B evaluation, not merely a way to make the model available.
- Experimentation platform purpose. Describe the role of an experimentation platform when evaluating a model online.
- Infrastructure effects on test outcomes. Explain how a model that performs well on paper or improves click-through rate can still worsen the product experience when deployed.
- Infrastructure guardrails in A/B tests. Describe why infrastructure metrics should be monitored as A/B test guardrails alongside user success metrics, and identify the kinds of regressions they can reveal.
- Lightweight controlled model experiments. Describe how an existing feature flag system can support a basic model experiment by randomizing exposure to model versions and logging results under a user or session ID. Explain that the aim is to isolate the model’s impact by routing traffic, measuring key metrics, and tracking users over time.
- Missing platform guardrails. Explain why a makeshift experiment needs cautious interpretation: without a real platform, guardrails such as automatic bucketing, holdouts, exposure logging, and sanity checks may be missing. State that this is a reason to accept more manual overhead, not to skip online testing altogether.
What it builds on
- Influencing online controlled experiments (chapter 1) · The relationship between offline evaluations and online controlled experiments.
- Offline metric categories (chapter 1) · The existence of different categories of offline metrics.
- What are AI model offline evaluations? (chapter 1) · The purpose of offline evaluations.
- Latency degradation experiments (chapter 4) · Latency degradation tests can quantify how latency affects product and user metrics.
- Why latency and load time really matter (chapter 4) · Why latency and load time affect user experience and business metrics.
Section by section
Introduction
The chapter introduction explains why offline evaluations are necessary but A/B tests are needed to judge a model’s effect on real users and product outcomes.
- Offline and online evaluation roles: Explain why offline evaluations are needed before launch but cannot establish a model’s real impact on users, and what an A/B test adds.
Why it matters: A promising offline result is not yet evidence that the model improves the user experience. - Model scores and product outcomes: Describe why an offline improvement or a better local metric may not translate into a better product experience, and why a modest offline gain can still matter online.
Why it matters: The useful question is whether the model makes the product better for users, not just whether one score rises.
Key terms: offline evaluations, online data, A/B test, online controlled experiments, offline metrics, user metrics, local metric, downstream experience, user behavior
6.1 A/B testing
The section introduces A/B testing as a way to measure a model’s impact with real users, beyond what offline evaluations can predict.
- Offline and online evidence: Explain why an A/B test is needed even after offline evaluation: offline results provide directional confidence, while an online experiment measures what happens when the model interacts with real users in the product.
Why it matters: It explains why promising offline results do not settle whether a model helps in production. - Test and control comparison: Describe an A/B test as measuring a change’s impact by comparing key metrics for a test group that receives the change with a control group that does not. State that the metrics should reflect meaningful user outcomes.
Why it matters: The comparison gives the experiment its basis for judging whether a model change made a meaningful difference.
Key terms: offline evaluations, directional confidence, online controlled experiment, A/B test, A/B testing, test group, control group, treatment, variant, key metrics
6.1.1 Why A/B test
This section explains why A/B tests are needed to measure a model’s real product impact rather than relying on offline predictions.
- Offline predictions and online evidence: Distinguish what offline evaluations predict from what an A/B test measures, and explain how results from a subset of users inform a possible launch to all users.
Why it matters: A strong offline result is a prediction, not proof of a better product. - Model impact within the product: Explain why offline results can differ from A/B test results and describe the kinds of product impact the test can reveal.
Why it matters: Testing in the product can reveal gains or degradations that offline results miss. - Evidence over expected success: Describe the A/B-testing mindset of separating hoped-for improvements from observed product outcomes.
Why it matters: This mindset keeps the experiment focused on learning rather than defending a promising idea.
Key terms: offline evaluation, A/B test, online, real user behavior, product experience, real product impact, A/B testing is a mindset, product better
6.2 The ‘right time’ to A/B test a model
This section frames A/B-test readiness as a decision based on model maturity, timelines, and offline evidence rather than a guarantee of online success.
- A/B Test Timing Factors: Explain how model maturity, product and engineering timelines, and confidence in offline evaluations affect when to run an online evaluation.
Why it matters: These factors keep the testing decision from resting on model performance alone. - Offline Exit Criteria: Describe the kinds of minimum offline conditions that make a model a candidate for online testing.
Why it matters: Exit criteria help a team decide whether a model is ready to be considered for a live experiment. - Limits of Offline Readiness: Distinguish being ready to test online from having proven that a model will succeed online.
Why it matters: This distinction prevents offline results from being mistaken for proof of product impact.
Key terms: model maturity, online evaluation, offline evaluation practices, offline exit criteria, baseline, primary offline metric, guardrail metrics, segment-level diagnostics, error analysis, offline evaluations
6.2.1 Model maturity
Model maturity determines whether an A/B test should explore an early idea’s viability or precisely evaluate a new version of an established model.
- Early-stage model testing strategy: Explain why an early-stage model should be tested for promise with real users rather than readiness for a full rollout, and describe the offline and A/B testing approach proposed here.
Why it matters: The question of viability calls for early evidence, not a test designed to justify exposing the model to everyone. - Mature model testing strategy: Describe how the A/B testing strategy changes when evaluating a new version of a mature model, including the question asked and the evidence and safeguards emphasized.
Why it matters: A change to an established model needs precise evidence of improvement without overlooking harm to existing performance.
Key terms: early-stage models, offline evaluation suite, feature-level metrics, model maturity, top-line metrics, feature level metrics, regressions
6.2.2 Timelines
This section explains how to time model A/B tests to get useful feedback without assuming every model needs the same testing schedule.
- Early iterative A/B testing: Explain why a team prioritizing speed might run lightweight A/B tests early in model development, particularly when exploring a new model type or product application.
Why it matters: An early test can challenge assumptions before the team invests in a polished launch. - Context-dependent testing timelines: Describe the factors that determine when and how thoroughly to A/B test a model, and how the timing affects the kind of signal a team needs.
Why it matters: The right test schedule is the one that produces a useful decision at an acceptable level of risk.
Key terms: lightweight, iterative A/B tests, minimum viable product (MVP)-style approach, directional signals, risk tolerance, offline signals, fail fast, sensitive, fast-moving metrics
6.2.3 How offline evaluations influence A/B test readiness
This section explains how the strength of offline evaluations changes the risk, timing, and safeguards of a model A/B test.
- Offline evidence and test risk: Explain how strong offline evaluations reduce the risk of entering an A/B test by providing directional confidence about model quality, failure modes, user cohorts, and likely product impact.
Why it matters: Offline evidence helps you judge what risk you are taking before exposing users to a model. - Weak evidence and online safeguards: Describe why an A/B test is not a substitute for understanding model behavior, and how to proceed more cautiously when offline evaluations are weak.
Why it matters: When you trade offline confidence for an earlier test, you need a way to notice and respond to unexpected harm. - Context-dependent offline evidence bar: Explain how model maturity and timeline constraints affect the amount of offline confidence sought before an A/B test.
Why it matters: A readiness decision should fit the purpose and maturity of the model rather than follow one fixed evidence threshold.
Key terms: offline evaluations, directional confidence, failure modes, segment-level behavior, higher-risk learning exercise, observability, real-time dashboards, alerts, fallbacks, model maturity
6.3 Designing an A/B test
Introduces model A/B test design as a comparison against an appropriate control, then frames design decisions as layers from logistics to core product value.
- Model variant and control: Explain what is introduced to users in a model A/B test and how the control differs for a brand-new model versus an updated one.
Why it matters: The comparison determines what effect the experiment can measure. - Layers of test design: Describe how the section organizes A/B test design decisions from outer layers to the center.
Why it matters: The layered view distinguishes logistical choices from decisions central to an experiment’s product value.
Key terms: A/B testing, control group, heuristic, model version, layers, surface-level logistics, core aspects
6.3.1 Operational setup
This section identifies the execution decisions for an A/B test and explains how exposure rules determine where and with whom to run it.
- A/B test execution decisions: Describe the operational decisions that specify how an A/B test runs: sample size, duration, traffic allocation, exposure rules, and whether assignment ramps up gradually or happens all at once.
Why it matters: These decisions define how the experiment will actually run in the product. - Strategic exposure targeting: Explain that exposure rules, also called targeting criteria, restrict the experiment to relevant users or contexts and can be used to choose a strategic starting audience.
Why it matters: Targeting keeps a test relevant to the change being evaluated and lets teams choose where to test first.
Key terms: sample size, duration, traffic allocation, exposure rules, ramp up, targeting criteria, device type, geographic region, user tenure
6.3.2 Experimental design
This section explains how an A/B test’s experimental design defines what changes, who receives each variant, and how model impact is measured and interpreted.
- Experimental design choices: Describe the middle layer of A/B test design: the treatment and baseline, unit of randomization, metrics, segmentation plans, and guardrails. Explain how these choices connect the hypothesis to the operational rollout.
Why it matters: Separating design choices from rollout choices helps make a model test interpretable. - Test and control variants: Explain what users in the test and control variants experience and identify the unit of randomization as a design choice.
Why it matters: The comparison only makes sense when you can say who receives the new model and what they are compared against. - Success and guardrail metrics: Distinguish success metrics from guardrail metrics and explain why an improvement in a primary metric does not, by itself, establish a good outcome.
Why it matters: Guardrails help catch costs that a strong headline result can hide. - Metric interpretation and test sizing: Explain why a metric change may need further analysis and how metric sensitivity informs the sample size or duration chosen for the rollout.
Why it matters: Interpreting a result requires both investigating what drove it and allowing enough data to detect a genuine effect.
Key terms: experimental design, unit of randomization, segmentation plans, operational rollout, test variant, control variant, targeting criteria, success metrics, guardrail metrics, user retention
6.3.3 Hypothesis and purpose
The section explains how an A/B test’s purpose shapes its hypothesis, success criteria, and decisions about rollout or further iteration.
- Experiment purpose and decision criteria: Explain why an A/B test needs a clear learning objective, and describe how its hypothesis, success criteria, and decision framework make that objective actionable.
Why it matters: A clear purpose helps you choose a coherent test and decide what to do with its results.
Key terms: hypothesis, success criteria, decision framework, exposure rules
6.3.4 Model purpose & intent
This section explains why an A/B test should be grounded in the model’s intended role and product impact, not just a numerical result.
- Model purpose and intent: Explain what the model’s core intent should clarify before interpreting an A/B test: the desired change in user behavior, the model’s role in the product, acceptable risk and trade-offs, and the assumptions the test might validate or challenge.
Why it matters: A test can pass numerically without showing that the model serves the product’s real purpose. - Intent driven test decisions: Describe how stating the model’s intent helps a team write hypotheses, choose metrics, interpret ambiguous results, and avoid wins that do not align with product goals.
Why it matters: The stated purpose gives the team a way to check whether a promising result is actually relevant to the product.
Key terms: model purpose & intent, intended user behavior, product role of the model, risk profile and acceptable trade-offs, assumptions, hypotheses, metrics, ambiguous results, false wins, product goals
6.3.5 Experiment validity checks before interpreting results
This section explains the checks needed to decide whether an A/B test’s results can be trusted as evidence of a model’s effect.
- Assignment versus actual exposure: Distinguish assignment to a variant from actually seeing the changed model output, and explain why logging assignment alone can misrepresent who received the treatment.
Why it matters: A treatment comparison is hard to interpret if some people counted as treated never saw the changed output. - Randomization and analysis safeguards: Describe the checks on random assignment, the observed versus planned traffic split, and decisions made before versus after examining results.
Why it matters: These checks help separate a planned, properly assigned comparison from a result shaped by execution problems or decisions made after seeing the data. - Serving and measurement integrity: Explain how to check what model behavior was served, whether other experiments or instrumentation affected the metrics, and why unresolved problems limit a launch decision.
Why it matters: Reliable metrics do not establish that the treatment caused a change unless you can verify what was served and how outcomes were measured.
Key terms: assignment, exposure, treatment variant, exposure logging, randomization, unit of randomization, sample ratio, pre-planned segments, stopping rules, model version logging
6.3.6 Movie recommendation A/B test example
The section applies an A/B test design to a proposed movie recommender update, from its product hypothesis and variants through its metrics, operational setup, and launch decisions.
- Contextual feature hypothesis: Explain why the team is testing contextual features in an existing recommender and what product benefit the change must deliver.
Why it matters: The test needs a reason to change a recommender that already works, not just a higher model score. - Success metrics and guardrails: Describe the hypothesis, metric roles, and launch criterion for the recommender test.
Why it matters: Separating gains from guardrails prevents an apparent engagement win from masking a worse experience. - Control and treatment variants: Distinguish the control and two treatments and explain what their comparison reveals.
Why it matters: These variants make the test informative about both adding context and choosing its strength. - Experiment assignment and preparation: Describe how users, exposure, duration, and sample size are specified before the test.
Why it matters: A defined operating plan makes the treatment comparison more consistent and interpretable. - Predefined launch decisions: Explain how the team would interpret wins, trade-offs, flat results, and segment gains before deciding what to do next.
Why it matters: A decision framework keeps an attractive single metric from overriding product value and serving constraints.
Key terms: contextual features, viewing intent, user embeddings, content embeddings, primary metrics, secondary metric, guardrail metrics, launch criterion, control, Treatment A
6.4 Prerequisites for running A/B tests
This introduction explains why A/B tests of AI models need preparation beyond that of traditional product experiments, without yet detailing the prerequisites.
- Model experiment complexity: Explain why an A/B test of an AI model can be harder to run reliably than an experiment changing a UX component, citing the risks named here.
Why it matters: These risks explain why running a model A/B test calls for preparation before launch.
Key terms: model behavior, inference cost, infra issues
6.4.1 Latency monitoring
This section explains how inference latency affects users and can confound A/B test results, making real-time latency monitoring an experiment guardrail.
- Latency and experiment interpretation: Explain what inference latency is and why a speed difference between treatment and control can make an A/B test misleading.
Why it matters: A change in product metrics may reflect response speed rather than the quality of the model’s predictions. - Latency as experiment guardrail: Describe how teams should monitor latency during a model A/B test and what comparisons the monitoring should support.
Why it matters: Segmented, real-time monitoring makes it easier to spot speed differences that could distort the experiment.
Key terms: inference latency, treatment model, control, confound, guardrail metric, real time, model version, treatment group
6.4.2 Feature drift monitoring
This section explains how feature drift can undermine a model A/B test and how to detect and respond to significant shifts in production inputs.
- Feature drift and test validity: Explain what feature drift is and why it can make an offline-tested model behave unpredictably during an A/B test.
Why it matters: A test result is harder to interpret if the model is receiving inputs unlike those on which it was trained. - Statistical feature drift detection: Describe how comparing training and production feature distributions can flag drift, including the roles of the KS test and PSI.
Why it matters: Distribution checks can reveal an input mismatch before it silently distorts the experiment. - Early drift monitoring response: Explain when to monitor key features during an A/B test and what significant drift should prompt investigators to consider.
Why it matters: Early alerts give the team a chance to investigate a population change before trusting the test result.
Key terms: feature drift, input features, training data, production data, Kolmogorov-Smirnov Test, KS test, Population Stability Index, PSI, drift monitoring, key features
6.4.3 Logging still matters
This section explains why model A/B tests need logs that connect experiment assignment to what users actually saw and did, so results can be interpreted reliably.
- Assignment to outcome logging: Describe the minimum data needed to interpret a model A/B test: variant assignment, actual exposure, model version, returned output, and subsequent user behavior.
Why it matters: That record lets you interpret an A/B result rather than relying on assignment alone. - Serving path attribution: Explain how fallbacks, caches, and safety layers can make the experience differ from the assigned model, and why logging these paths matters when interpreting results.
Why it matters: Serving-path logs help determine whether a metric change is attributable to the model and trustworthy enough for a launch decision.
Key terms: experiment assignment, exposure events, model version, model output, downstream user outcomes, serving path, fallback path, cache, safety layer, latency
6.5 Quirks of A/B testing models
This section explains why AI model A/B tests need more careful interpretation than typical UX experiments.
- Model A/B test complexity: Explain why an AI model A/B test cannot be interpreted exactly like a typical UX feature test, and what can go wrong if it is.
Why it matters: It helps you avoid treating an unclear model experiment as a straightforward verdict on whether the model works.
Key terms: cold start effects, high variances, training data bias, A/B test
6.5.1 Multiple model variants increase complexity
This section explains why testing multiple model variants demands more traffic, creates false-positive risk, and requires a clear hypothesis for each variant.
- Traffic needs for multiple variants: Explain why adding model variants increases the traffic or duration needed for an A/B test.
Why it matters: A test with too many variants may exceed the traffic or time available for reliable decisions. - False positives across comparisons: Explain the multiple-comparisons problem in a multi-variant experiment and describe the corrections the section names.
Why it matters: Without accounting for multiple comparisons, an apparently successful variant may be a false positive. - Purposeful variant selection: Justify giving every tested variant a clear hypothesis and a plan for interpreting its results.
Why it matters: A variant is useful only if its results can answer a question the team intended to test.
Key terms: treatment variant, sample size, exposure, multiple-comparisons problem, false positive, Bonferroni correction, false discovery rate (FDR) correction, hypothesis, test variant, interactions
6.5.2 Novelty effect
Explains how initial user curiosity can inflate A/B test results and how to check whether an apparent lift lasts.
- Novelty Driven Metric Lift: Explain how a changed experience can temporarily raise treatment metrics even when the model is not better.
Why it matters: An early metric win may reflect curiosity rather than a lasting improvement. - Time Based Lift Check: Describe how to use experiment duration and week-over-week primary-metric trends before deciding to launch.
Why it matters: Checking the trend over time reduces the chance of shipping on a temporary spike.
Key terms: novelty effect, treatment metrics, early lift, primary metric, week-over-week, experiment window
6.5.3 Model warm-up and cold start effects
This section explains how model warm-up and cold-start effects can distort early A/B test results and how to plan when primary analysis should begin.
- Cold-start effects on results: Explain why early A/B test outcomes may reflect model or system stabilization rather than long-term model quality, including for non-adaptive models.
Why it matters: An early dip in engagement may say more about startup conditions than about the model being tested. - Predefined warm-up analysis criteria: Describe how to set warm-up criteria before an A/B test and use them to separate stabilization from primary analysis while preserving enough post-warm-up data for comparison.
Why it matters: Setting the cutoff in advance avoids choosing it to suit the results and leaves a usable sample for evaluating the treatment.
Key terms: warm-up, cold-start behavior, treatment, primary analysis, minimum stabilization period, warm-up period, interaction volume, stability-based
6.5.4 Variance and sensitivity to noise
Explains why noise can make AI A/B test results unstable and how experiment design can improve the ability to detect a model’s effect.
- High variance in test outcomes: Explain what high variance means in an AI A/B test and why it makes the model’s effect difficult to interpret.
Why it matters: A noisy result can mislead you about whether a model actually improves the experience. - Detectability through experiment design: Describe which conditions make meaningful differences harder to detect and which experiment-design choices the section recommends in response.
Why it matters: Design choices affect whether a test can reveal a meaningful model impact rather than leave it buried in noise.
Key terms: high variance, observed effect sizes, control experience, false positives or negatives, sample size, stable metrics, robust evaluation windows, small effect sizes, long feedback loops, rare conversions
6.5.5 Offline and online mismatch
This section explains why strong offline results can fail to translate into A/B test gains and how to investigate the disagreement.
- Offline and product goal alignment: Explain why a high offline metric does not guarantee a successful A/B test: the metric or its labels may not reflect the user behavior the product actually values.
Why it matters: It keeps a strong offline score from being mistaken for proof of product value. - Sources of offline online mismatch: Describe how stale or unrepresentative data and differences in production features or serving conditions can make offline performance misleading.
Why it matters: These differences give concrete places to look when an offline win disappears online. - Investigating conflicting evaluation results: Describe how to treat an offline–online disagreement as a diagnostic starting point rather than assuming one cause, using A/B test evidence to examine segments, logs, inputs, exposure, and the offline evaluation design.
Why it matters: A structured investigation helps distinguish a flawed offline measure from problems in live delivery or targeting.
Key terms: offline metric, real product objective, A/B test, stale, training data, features, production, diagnostic starting point, segments, logs
6.6 Interpreting ambiguous or low-signal results
The section explains how to interpret ambiguous AI-model A/B test results without overstating the evidence and how to use them to make a launch, iteration, or stop decision.
- Limits of low-signal results: Distinguish an experiment that cannot detect an effect from evidence that the model has no effect. Explain how too few users, too short a test, rare outcomes, or a highly variable metric can make results inconclusive.
Why it matters: This keeps an inconclusive test from being mistaken for a definitive verdict on the model. - Small-effect launch tradeoffs: Explain why a genuinely small positive effect is not automatically a launch decision. Weigh its product value against the new model’s complexity, cost, latency, reliability, and operational burden.
Why it matters: A measurable gain can still be a poor product decision if it is expensive or difficult to sustain. - Metrics matched to model purpose: Explain how an A/B test can miss a model’s intended benefit when its metric measures the wrong outcome. Describe why purpose and success criteria should be set before the test rather than reinterpreted afterward.
Why it matters: A result is interpretable only in relation to the outcome the model was meant to improve. - Planned versus post-hoc segments: Describe how an overall A/B test result can conceal different effects for different users or contexts. Distinguish the evidential weight of segments planned before the test from segments discovered after examining results.
Why it matters: This allows useful subgroup signals to be investigated without overclaiming a pattern found after the fact. - Evidence-based experiment decisions: Describe how to return to the original hypothesis and decision framework when results are ambiguous. Check validity and exposure, consider primary and guardrail metrics alongside offline evaluations, and judge whether the effect matters before deciding to launch, iterate and retest, or stop.
Why it matters: An inconclusive test can still support a better decision when its evidence and limitations are assessed systematically.
Key terms: low-signal result, underpowered, random variation, model effect, operational risk, infrastructure cost, model’s purpose, success criteria, short-term click-through metric, segment analysis
6.7 Engineering considerations
This section identifies infrastructure and an experimentation platform as the two key engineering considerations for evaluating a model in an online A/B test.
- Infrastructure as evaluation concern: Explain why model infrastructure is part of an online A/B evaluation, not merely a way to make the model available.
Why it matters: A model that works in isolation may still cause problems when served in the product. - Experimentation platform purpose: Describe the role of an experimentation platform when evaluating a model online.
Why it matters: The platform lets you assess the model’s effect in the product rather than relying only on offline results.
Key terms: infrastructure reliability, latency, evaluation strategy, experimentation platform, control, test variant, offline evaluations
6.7.1 Infrastructure-induced latency or errors
Infrastructure performance belongs in model A/B test evaluation because a model can improve a success metric while harming users, system health, or cost.
- Infrastructure effects on test outcomes: Explain how a model that performs well on paper or improves click-through rate can still worsen the product experience when deployed.
Why it matters: A lift in one metric is not a complete win if deploying the model worsens the experience behind it. - Infrastructure guardrails in A/B tests: Describe why infrastructure metrics should be monitored as A/B test guardrails alongside user success metrics, and identify the kinds of regressions they can reveal.
Why it matters: Guardrails make the operational cost of an apparent A/B test win visible before a shipping decision.
Key terms: latency, instability, scaling issues, engagement, success metrics, guardrail metrics, inference latency, system throughput, CPU and GPU utilization, cost per request
6.7.2 If you just don't have an A/B testing platform
This section explains how to start controlled online model testing without a dedicated A/B testing platform, using lightweight infrastructure or a hosted service while accounting for missing guardrails.
- Lightweight controlled model experiments: Describe how an existing feature flag system can support a basic model experiment by randomizing exposure to model versions and logging results under a user or session ID. Explain that the aim is to isolate the model’s impact by routing traffic, measuring key metrics, and tracking users over time.
Why it matters: It gives you a practical way to test a model online before investing in a full platform. - Hosted experimentation service trade-offs: Explain how a hosted experimentation service can provide randomization, assignment, exposure logging, and metrics pipelines without an in-house build, while still requiring work to integrate with data and model-serving systems.
Why it matters: A hosted service is another route to testing, but its integration cost belongs in the decision. - Missing platform guardrails: Explain why a makeshift experiment needs cautious interpretation: without a real platform, guardrails such as automatic bucketing, holdouts, exposure logging, and sanity checks may be missing. State that this is a reason to accept more manual overhead, not to skip online testing altogether.
Why it matters: Recognizing the missing safeguards helps you test pragmatically without overstating what the results show.
Key terms: feature flag system, randomize exposure, user or session ID, controlled way, experimentation-as-a-service platform, randomization, assignment, exposure logging, metrics pipelines, guardrails
6.8 Summary
Where this is used later
- Choosing metrics that actually matter (chapter 8) · Examines the choice of metrics that reflect meaningful product outcomes.
- (chapter intro) (chapter 7) · The basic purpose of A/B testing models.
- The case for LLM-as-a-judge (chapter 9) · Why A/B tests are used to evaluate models.
- Model selection playbook (chapter 7) · Offline evaluations inform whether a model is ready for an A/B test.
- Internal beta testing before exposing to users (chapter 7) · Assumes the distinction between offline evidence and readiness for an A/B test.
- Feedback loops unique to AI models (chapter 8) · the comparison of treatment and control in an A/B test
- The blind spots of online metrics (chapter 8) · Provides the distinction between headline measures and important outcomes they may miss.
- Interim checks (chapter 7) · Interpreting experiment results only after appropriate validity checks
Concepts explained in this chapter
- A/B testA controlled online experiment that compares outcomes between two variants, typically a control and a treatment, using assigned users or traffic.
- offline evaluationA diagnostic evaluation performed on non-live data to reveal model failures or unintended optimization behavior.
- offline metricAn offline metric is a quantitative measure of model or algorithm behavior computed from historical or otherwise non-live evaluation data.
- online evaluationEvaluation of an AI system in a live or production environment using real-time system behavior or user interactions.
- online experimentationA live controlled evaluation of a model or product change using real users or production traffic to measure outcomes.
- variantA test variant, or treatment, is an experimental condition that receives the change being evaluated.
- unit of randomizationA unit of randomization is the entity, such as a user, session, or account, that is independently assigned to an experiment variant.
- control groupA control variant is the reference condition in an experiment against which treatment variants are compared.
- treatment variantA model variant is an alternative model configuration or experimental treatment evaluated relative to a control or other variants.
- exposure loggingThe recording of whether an assigned user or unit actually received or experienced the changed model output.
- randomizationThe assignment of experiment units to variants by chance according to a specified allocation procedure.
- varianceVariance is a measure of how much a set of observed outcomes fluctuates around its expected or average value.
- cold-start behaviorThe behavior of a recommendation system when users or items have little or no interaction history.
- model maturityModel maturity is the stage of development and operational experience of an AI model, from early experimentation to established production use.
- data driftFeature drift detection is the identification of changes over time in the distributions or behavior of model input features.
- user experienceUser experience is the overall experience a user has when interacting with a product, including its usefulness, quality, and responsiveness.
- impactProduct impact is the effect that a model or product change has on product outcomes such as retention, engagement, or satisfaction.
- test suiteA collection of evaluations that are run regularly to assess a model across defined criteria or scenarios.
- content discovery rateContent discovery rate measures how many new titles or genres users try within a defined period.
- contextA contextual feature is an input variable describing the circumstances surrounding a user's current interaction or request.
- minimum detectable effectThe minimum detectable effect is the smallest true change in an outcome that an experiment is designed to detect with specified statistical power and significance.
- power analysisPower analysis is a statistical procedure for determining the sample size needed to detect an effect under specified assumptions.
- language modelA neural language model trained on large text corpora to understand, generate, or transform natural-language content.
- decision frameworkA decision framework is a structured set of criteria and reasoning rules for choosing among actions based on evidence.
- evaluation criterionA primary metric, or success metric, is the main outcome measure selected to evaluate whether an experiment achieves its intended objective.
- evaluation designAn evaluation design is a deliberate specification of the data, metrics, judgments, comparisons, scenarios, and decision rules used to assess a model or AI system.
- latencyThe total elapsed time from receiving a user request to displaying the complete response, including all intermediate processing steps.
- segmented evaluationSegment analysis compares evaluation results across defined subsets of users or data to reveal differences hidden by aggregate results.
- guardrail metricA guardrail metric is a metric monitored to ensure that an experiment does not cause unacceptable harm while optimizing its primary objective.
- sample sizeSample size is the number of observations or participants included in a statistical experiment.
- population stability indexThe Population Stability Index measures the degree of distribution shift between two datasets or populations.
- infrastructureInfrastructure is the computing, storage, networking, and runtime support required to operate a software system.
- hypothesisA hypothesis is a testable statement about a population or experimental effect evaluated using statistical evidence.
- continuous improvementA dynamic process in which a model's predictions influence user behavior and that behavior changes the data observed by the model later.
- metricA quantitative measure of model performance derived from behavior or outcomes observed during live operation.
- observabilityThe extent to which a system's internal state can be inferred from its outputs, logs, and other external signals.
- costCost measures the computational, financial, or resource expenditure required to operate a model or system.
- memory usageMemory footprint is the amount of memory occupied by a model or software process during operation.
- regressionA regression is an unintended deterioration in a model or system behavior after a change.
- resource utilizationHardware utilization is the proportion of available processor capacity consumed by a running system.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
