Chapter 7: From offline evaluation to live experiment
Positions A/B testing as one part of a broader model evaluation system that connects offline evaluation with online experimentation.
After this chapter you will be able to
- A/B tests in evaluation. Explain why an A/B test should be viewed as part of a broader AI model evaluation system rather than as a standalone tactic.
- Purpose of the selection playbook. Explain why a model selection playbook helps teams choose variants for A/B tests without making selection unnecessarily bureaucratic.
- Playbook evidence and selection rubric. Describe the core pieces of the playbook and how they support a decision about which variants graduate to an A/B test.
- Pretest questions and decision rule. State the practical questions a lightweight playbook should answer before a variant enters an A/B test.
- Internal beta purpose and limits. Explain why teams run an internal beta before a user-facing A/B test and why its results cannot establish product impact.
- Structured employee feedback process. Describe how to expose employees to model variants and collect feedback that can inform the next testing decision.
- Popularity driven ranking bias. Trace how the movie recommendation beta exposed a repeated top-ranked title and how the team could diagnose and address it.
- Multidimensional model review. Explain how the rubric consolidates offline metrics, product-aligned heuristics, model maturity, strategic alignment, operational readiness, and sandbox or beta feedback for a final review before an A/B test.
- Explicit model trade-offs. Describe how reviewers use the rubric to compare model variants across dimensions, discuss trade-offs explicitly, and document a collaborative decision to advance a model.
- Maturity and selection thresholds. Explain how model maturity changes the purpose of an A/B test and why playbook thresholds also depend on the use case, product domain, company strategy, and innovation potential.
- Flexible trade-offs and firm gates. Distinguish trade-offs an early-stage LLM playbook might relax from requirements it should strengthen or preserve.
- Explicit advancement decisions. Describe what a team should agree and record when advancing a model despite known gaps.
- Stakeholder alignment before launch. Explain why strong model results are not enough if stakeholders have different expectations about the model’s objective, behavior, or operational constraints.
- Offline risks as live watchpoints. Explain how findings from offline evaluations should shape monitoring during an A/B test: identify possible failure modes or edge cases, watch for them in live data, and set alert thresholds.
- Offline-to-online watchpoint workflow. Explain how to turn an offline failure mode into a live watchpoint by defining an equivalent online measurement and an alert threshold tied to the known gap.
- Proactive watchpoint interpretation. Explain what a team can do when a known risk appears online and what it can learn if the risk does not appear.
- Metric mirror table purpose. Explain how a metric mirror table links each key offline metric to an online watchpoint, a production computation, and an alert threshold. Describe how that link supports traceability, consistency, and actionability during an experiment.
- Online precision proxy limits. Distinguish offline Precision@10 from its online click-based watchpoint, and explain why the latter is a proxy rather than an identical replacement.
- Rolling experiment record. Describe how to build a rolling dataset that pairs each model variant’s offline evaluation scores with its A/B test outcomes and relevant experiment context. Explain why consistent metric definitions and a disciplined recording process matter.
- Correlation measures and meanings. Explain what positive, negative, and near-zero correlations say about how offline and online metrics move together. Distinguish Pearson’s linear, magnitude-sensitive measure from Spearman’s rank-based measure of a directionally consistent relationship.
- Correlation guided model selection. Describe how a correlation matrix summarizes relationships between offline and online metrics, and explain how repeated results should change the weights assigned to offline metrics in a model selection rubric.
- Limits of early results. Explain why early topline results should not determine whether a model A/B test succeeds or ends.
- Model-specific interim risk checks. Describe what interim checks should look for and how their findings should be used.
- Operational engineering support. Explain why infrastructure and tooling matter when moving from offline evaluation to online evaluation: they make the model selection playbook operational and help preserve offline insights.
- Live model metric coverage. Describe how live monitoring should track product and business metrics alongside the model-specific metrics used in selection and offline evaluation, computing them from live logs.
- Offline production formula parity. Explain why production and offline versions of a metric must use matching formulas, including category mappings, filters, and aggregation windows.
- Side-by-side comparison workflow. Describe how a comparison tool evaluates candidate models on the same dataset and makes their outputs and differences inspectable.
- Reproducible and blinded reviews. Justify keeping the comparison tied to evaluation data and concealing model identities while collecting feedback.
- Evidence-driven selection infrastructure. Explain how a rolling dataset and automated correlation updates help teams improve model selection over time.
- Connected evaluation decisions. Describe why moving from offline evaluation to live experimentation is an ongoing set of connected decisions rather than a one-time handoff.
What it builds on
- A/B testing (chapter 6) · The basic purpose of A/B testing models.
- How offline evaluations influence A/B test readiness (chapter 6) · Offline evaluations inform whether a model is ready for an A/B test.
- Combining evaluations to bridge the gap (chapter 2) · Different kinds of offline evaluation can be combined to inform development decisions.
- Hypothesis and purpose (chapter 6) · An A/B test needs a hypothesis and purpose.
- Evaluating bias in movie recommendations (chapter 3) · Assumes recommendations can be examined for systematic bias.
- Diagnosing model behavior (chapter 3) · Uses offline diagnosis of model behavior to identify failure modes.
- Experiment validity checks before interpreting results (chapter 6) · Interpreting experiment results only after appropriate validity checks
Section by section
Introduction
Positions A/B testing as one part of a broader model evaluation system that connects offline evaluation with online experimentation.
- A/B tests in evaluation: Explain why an A/B test should be viewed as part of a broader AI model evaluation system rather than as a standalone tactic.
Why it matters: This perspective helps connect online experiments with the rest of model evaluation.
Key terms: A/B testing, evaluation system
7.1 Model selection playbook
This section explains why teams need a practical playbook for choosing which model variants are ready for an A/B test and what that playbook should contain.
- Purpose of the selection playbook: Explain why a model selection playbook helps teams choose variants for A/B tests without making selection unnecessarily bureaucratic.
Why it matters: A clear but appropriately lightweight process can save testing capacity and prevent costly misalignment. - Playbook evidence and selection rubric: Describe the core pieces of the playbook and how they support a decision about which variants graduate to an A/B test.
Why it matters: Combining evaluation results with internal feedback and product considerations makes variant selection more defensible. - Pretest questions and decision rule: State the practical questions a lightweight playbook should answer before a variant enters an A/B test.
Why it matters: These questions turn evaluation findings into a testable hypothesis, a risk-monitoring plan, and a basis for action.
Key terms: model selection playbook, A/B test capacity, product goals, offline evaluation summary, sandbox or beta testing, model selection rubric, product heuristics, online watchpoints, decision rule, known trade-offs
7.2 Internal beta testing before exposing to users
This section explains how a structured internal beta can catch model and product failures before a user-facing A/B test, while remaining only a readiness signal rather than proof of user impact.
- Internal beta purpose and limits: Explain why teams run an internal beta before a user-facing A/B test and why its results cannot establish product impact.
Why it matters: This distinction helps you use employee feedback to reduce risk without mistaking it for evidence of customer impact. - Structured employee feedback process: Describe how to expose employees to model variants and collect feedback that can inform the next testing decision.
Why it matters: A guided process turns scattered opinions into patterns the team can act on before exposing users. - Popularity driven ranking bias: Trace how the movie recommendation beta exposed a repeated top-ranked title and how the team could diagnose and address it.
Why it matters: The example shows how a plausible-looking popular recommendation can hide a systematic ranking failure and distort later engagement signals. - Broader product readiness checks: Explain why internal testing should involve stakeholders using the product and look beyond model-output quality.
Why it matters: A model can produce good outputs yet still be difficult or unsafe to deploy in the actual product setting.
Key terms: internal beta test, readiness signal, product impact, internal application, internal use-only A/B test, feature flag, structured feedback, retrieval stage, re-ranking stage, global popularity feature
7.3 Model selection scoring rubric
The section presents a scoring rubric for bringing model-selection evidence together and documenting trade-offs before an A/B test.
- Multidimensional model review: Explain how the rubric consolidates offline metrics, product-aligned heuristics, model maturity, strategic alignment, operational readiness, and sandbox or beta feedback for a final review before an A/B test.
Why it matters: A shared view of the evidence makes it easier to explain why a model should advance. - Explicit model trade-offs: Describe how reviewers use the rubric to compare model variants across dimensions, discuss trade-offs explicitly, and document a collaborative decision to advance a model.
Why it matters: Explicit trade-offs let reviewers defend a choice even when neither model wins on every dimension.
Key terms: scoring rubric, offline performance metrics, product-aligned heuristics, model maturity, operational readiness, sandbox/beta feedback, trade-offs, model comparisons, decision to advance
7.4 Model maturity nuances in selection decisions
This section explains how model maturity and product context shape selection thresholds, especially for a first LLM feature, and how teams should make exceptions explicit before a live test.
- Maturity and selection thresholds: Explain how model maturity changes the purpose of an A/B test and why playbook thresholds also depend on the use case, product domain, company strategy, and innovation potential.
Why it matters: The same selection gate need not fit every model or product situation. - Product-specific LLM maturity: Explain why a company's first LLM-based product experience should usually be treated as early-stage, even when the underlying model family has succeeded elsewhere.
Why it matters: A proven model family does not remove uncertainty about a new product experience. - Flexible trade-offs and firm gates: Distinguish trade-offs an early-stage LLM playbook might relax from requirements it should strengthen or preserve.
Why it matters: Experimenting with a promising feature need not mean accepting avoidable safety or reliability risks. - Explicit advancement decisions: Describe what a team should agree and record when advancing a model despite known gaps.
Why it matters: An explicit decision and response plan makes a flexible selection defensible during a live experiment.
Key terms: model maturity, variant selection strategy, test-to-learn, playbook, pre-test gates, LLM-based product experience, early-stage, latency, cost, coverage
7.5 Stakeholder alignment and communication
The section explains how early stakeholder exposure and a documented model-selection decision can prevent misalignment before an A/B test.
- Stakeholder alignment before launch: Explain why strong model results are not enough if stakeholders have different expectations about the model’s objective, behavior, or operational constraints.
Why it matters: Agreeing on the details early can prevent a promising model from becoming a surprise at launch. - Leadership exposure during beta: Describe how including decision-makers in internal beta testing can build confidence before users see the model, and why their feedback needs a process.
Why it matters: Early, structured feedback can surface concerns without turning model development into a cycle of reactive changes. - Rubric as decision narrative: Explain how the model selection scoring rubric helps communicate a recommendation, and why introducing a new framework mid-development can be counterproductive.
Why it matters: A shared decision record helps stakeholders understand why a variant is moving forward rather than seeing only its final metrics.
Key terms: stakeholder misalignment, model selection playbook, model’s objective, online monitoring plan, internal beta testing, beta testing cohort, feedback, thrashing, model selection scoring rubric, offline evaluation results
7.6 Mapping offline insights into targeted online monitoring
This section explains how offline evaluation findings can guide monitoring during a live experiment and improve future model-selection decisions.
- Offline risks as live watchpoints: Explain how findings from offline evaluations should shape monitoring during an A/B test: identify possible failure modes or edge cases, watch for them in live data, and set alert thresholds.
Why it matters: Targeted monitoring gives the team a chance to notice known risks while there is still time to respond. - Offline and online metric learning: Explain why teams should link offline and online results over time: to find which offline metrics predict live performance and refine decisions for future experiments.
Why it matters: Learning which offline signals hold up with real users makes future experiment decisions better informed.
Key terms: offline evaluations, failure modes, edge cases, watchpoints, thresholds, alerts, offline results, online results, offline metrics, online performance
7.6.1 Translating offline evaluations into online watch points
This subsection shows how to turn weaknesses found offline into targeted live watchpoints for an A/B test, so teams can detect known risks early and see whether fixes hold up.
- Offline-to-online watchpoint workflow: Explain how to turn an offline failure mode into a live watchpoint by defining an equivalent online measurement and an alert threshold tied to the known gap.
Why it matters: A targeted watchpoint can surface a known weakness before the experiment’s final results arrive. - Batman repeat-title watchpoint: Describe how the Batman ranking issue becomes a segment-level repeat-title watchpoint, including what it measures and when it raises an alert.
Why it matters: The example makes a model-behavior concern measurable in live recommendations. - Proactive watchpoint interpretation: Explain what a team can do when a known risk appears online and what it can learn if the risk does not appear.
Why it matters: Watchpoints help teams act before a known issue distorts experiment results and learn from risks that never recur.
Key terms: failure modes, online watchpoints, primary experiment success metrics, alert thresholds, popularity bias, repeat-title frequency, top-N recommendations, user segment, control, watchpoints
7.6.2 Building a metric mirror table
This section shows how a metric mirror table connects offline evaluations to online monitoring while making computation rules, alert thresholds, and proxy relationships explicit.
- Metric mirror table purpose: Explain how a metric mirror table links each key offline metric to an online watchpoint, a production computation, and an alert threshold. Describe how that link supports traceability, consistency, and actionability during an experiment.
Why it matters: The table gives data science and engineering teams a shared basis for monitoring and acting on experiment results. - Production monitoring design choices: Describe the implementation checks the section gives for a mirror table: match offline and online computation rules, choose an aggregation cadence suited to detection, and set thresholds using both offline benchmarks and historical production variability.
Why it matters: These choices help alerts reflect meaningful changes rather than calculation differences or ordinary variation. - Online precision proxy limits: Distinguish offline Precision@10 from its online click-based watchpoint, and explain why the latter is a proxy rather than an identical replacement.
Why it matters: Recognizing the proxy relationship prevents a click-based watchpoint from being mistaken for a direct measurement of offline precision.
Key terms: metric mirror table, offline metric, online watchpoint, thresholds, formula parity, aggregation cadence, offline benchmarks, historical production variability, Precision@10, proxy
7.6.3 Tracking correlation between offline and online metrics over time
This section shows how to track the relationship between offline scores and online experiment outcomes over time, then use that evidence to improve model selection.
- Rolling experiment record: Describe how to build a rolling dataset that pairs each model variant’s offline evaluation scores with its A/B test outcomes and relevant experiment context. Explain why consistent metric definitions and a disciplined recording process matter.
Why it matters: A consistent experiment history lets you check predictive claims instead of assuming an offline metric forecasts product results. - Correlation measures and meanings: Explain what positive, negative, and near-zero correlations say about how offline and online metrics move together. Distinguish Pearson’s linear, magnitude-sensitive measure from Spearman’s rank-based measure of a directionally consistent relationship.
Why it matters: Choosing and interpreting a correlation measure helps you judge whether an offline signal has tracked an online outcome. - Correlation guided model selection: Describe how a correlation matrix summarizes relationships between offline and online metrics, and explain how repeated results should change the weights assigned to offline metrics in a model selection rubric.
Why it matters: The matrix connects accumulated experiment evidence to practical model selection decisions. - Complementary overlapping offline metrics: Explain why offline metrics that tend to move together can still be worth keeping, using precision@10 and recall@10 as the section’s example.
Why it matters: Keeping complementary measures can expose trade-offs that one offline score hides.
Key terms: rolling dataset, offline evaluation scores, online experiment outcomes, metric definitions, correlation, Pearson’s r, Spearman’s ρ, linear relationship, rank-based monotonic relationship, correlation matrix
7.7 Interim checks
Explains how to use interim experiment data to catch model-specific risks without treating volatile early results as a final readout.
- Limits of early results: Explain why early topline results should not determine whether a model A/B test succeeds or ends.
Why it matters: An early swing in a metric is not necessarily evidence of the model’s effect. - Model-specific interim risk checks: Describe what interim checks should look for and how their findings should be used.
Why it matters: These checks can reveal a reason to intervene while preserving a sound final assessment of business outcomes.
Key terms: interim checks, topline outcomes, false positive rate, guardrail-first focus, segment drilldowns, feature contribution shifts, topline readout
7.8 Engineering considerations
This section explains why engineering infrastructure and tooling are needed to put the model selection playbook into practice and carry offline evaluation insights into online evaluation.
- Operational engineering support: Explain why infrastructure and tooling matter when moving from offline evaluation to online evaluation: they make the model selection playbook operational and help preserve offline insights.
Why it matters: Engineering support is what lets evaluation insights inform a live process rather than remain a plan.
Key terms: model selection playbook, offline evaluation stages, online evaluation phase, infrastructure and tooling
7.8.1 Realtime monitoring tied to product and business metrics
This section explains how to monitor live product, business, and model-specific metrics in a way that remains comparable with offline evaluation and alerts teams to known risks.
- Live model metric coverage: Describe how live monitoring should track product and business metrics alongside the model-specific metrics used in selection and offline evaluation, computing them from live logs.
Why it matters: Monitoring the same model measures in production makes offline findings visible after launch. - Offline production formula parity: Explain why production and offline versions of a metric must use matching formulas, including category mappings, filters, and aggregation windows.
Why it matters: Matching calculations lets teams interpret live results against their offline baseline. - Metric mirror alert thresholds: Describe how alert thresholds from the metric mirror table provide early warnings when risks identified offline appear in live data.
Why it matters: Alerts help teams notice an anticipated model problem as it emerges in production.
Key terms: model-specific metrics, real-time metric pipelines, live logs, formula parity, category mappings, filters, aggregation windows, alerting thresholds, metric mirror table, offline risks
7.8.2 Side-by-side model comparison tools
This section explains how internal side-by-side tools can make model comparisons reproducible and inspectable before an A/B test, while reducing bias in human feedback.
- Side-by-side comparison workflow: Describe how a comparison tool evaluates candidate models on the same dataset and makes their outputs and differences inspectable.
Why it matters: Comparing the same cases makes model differences easier to inspect before a live experiment. - Segment views and structured feedback: Explain how filtering and inline annotations help reviewers examine model differences that an aggregate view might miss.
Why it matters: These features turn specific observations into feedback without hiding differences across groups. - Reproducible and blinded reviews: Justify keeping the comparison tied to evaluation data and concealing model identities while collecting feedback.
Why it matters: A repeatable comparison is more useful for selection, and blind review makes qualitative feedback more reliable.
Key terms: batch evaluation pipeline, side-by-side visual diffing, raw predictions, derived heuristics, segment filtering, annotation and feedback collection, structured feedback set, model selection rubric, anonymize model labels, randomize the display order
7.8.3 Reliable infrastructure for iterative improvement
Reliable infrastructure connects offline results, online outcomes, and decision history so each live experiment improves future model selection.
- Evidence-driven selection infrastructure: Explain how a rolling dataset and automated correlation updates help teams improve model selection over time.
Why it matters: It makes the selection playbook responsive to evidence from past experiments rather than leaving it as a static checklist. - Connected evaluation decisions: Describe why moving from offline evaluation to live experimentation is an ongoing set of connected decisions rather than a one-time handoff.
Why it matters: Connecting these decisions allows an experiment's results to inform the next model selection decision.
Key terms: rolling dataset, correlation updates, metric weights, selection rubric, historical predictive power, connected decisions, A/B testing, model evaluation lifecycle, decision history
7.9 Summary
Where this is used later
- (chapter intro) (chapter 8) · The connection between offline insights and online monitoring.
- The blind spots of online metrics (chapter 8) · Translating offline evaluation findings into online watch points.
- Fairness across segments (chapter 8) · Connects offline evaluation findings to targeted online monitoring.
- How to combat feedback loops in evaluations (chapter 8) · Tracking correlation between offline and online metrics over time
- Revisiting metrics as models and products evolve (chapter 8) · Tracking correlation between offline and online metrics over time.
- Monitoring for metric drift in retraining pipelines (chapter 8) · Tracking offline and online metrics over time.
Concepts explained in this chapter
- model selection playbookA model selection playbook is a structured process for comparing model variants and documenting the evidence, trade-offs, and readiness checks behind a selection decision.
- watchpointA live monitoring check that tracks specified online behavior, model weaknesses, or risks and triggers investigation or intervention when needed.
- internal beta testingStructured evaluation of a model or product change by employees before exposure to external users.
- position biasPosition bias is the tendency for a response's displayed position in an evaluation prompt to affect the judge's verdict independently of the response content.
- rerankingRe-ranking is the process of reordering an initially retrieved or recommended list using additional criteria or signals.
- two-stage architectureA recommendation architecture that first generates candidate items and then ranks a smaller set using additional signals.
- rubricA defined set of criteria and scoring guidance for assessing model outputs.
- beta testingPre-release testing in which selected users evaluate a product or model to identify issues before wider release.
- correlationCorrelation is a statistical association between variables that does not by itself establish that one variable causes another.
- rolling datasetA rolling dataset is a continuously maintained collection of records that incorporates newly available evaluation and operational results over time.
- correlation matrixA correlation matrix is a tabular array whose cells contain pairwise correlation values for variables represented by its rows and columns.
- online-offline correlationOffline-to-online correlation is the analysis of how changes in offline evaluation metrics relate to outcomes observed in live experiments.
- model maturityModel maturity is the stage of development and operational experience of an AI model, from early experimentation to established production use.
- offline metricAn offline metric is a quantitative measure of model or algorithm behavior computed from historical or otherwise non-live evaluation data.
- failure modeA failure mode is a recurring way in which a model or system fails under particular conditions.
- hypothesisA hypothesis is a testable statement about a population or experimental effect evaluated using statistical evidence.
- treatment variantA model variant is an alternative model configuration or experimental treatment evaluated relative to a control or other variants.
- metricA quantitative measure of model performance derived from behavior or outcomes observed during live operation.
- thresholdA threshold is a predefined cutoff value used to determine whether a metric or evaluation result satisfies a requirement.
- proxy metricA proxy metric is an indirect measure whose value is intended to indicate a target quality or outcome that is difficult to measure directly.
- edge caseAn edge case is an uncommon or boundary-condition input or situation that can expose unusual model behavior.
- monitoringThe continuous observation of model behavior, system signals, and user outcomes during live operation.
- offline evaluationA diagnostic evaluation performed on non-live data to reveal model failures or unintended optimization behavior.
- diversityA diversity index is a quantitative measure summarizing the variety of items, categories, or outcomes in a set.
- percentileA latency percentile is a value below which a specified proportion of observed system response times falls.
- infrastructureInfrastructure is the computing, storage, networking, and runtime support required to operate a software system.
- online evaluationEvaluation of an AI system in a live or production environment using real-time system behavior or user interactions.
- accuracyA measure of how often a model's outputs match desired or correct outcomes.
- data driftFeature drift detection is the identification of changes over time in the distributions or behavior of model input features.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
