A/B test
An A/B test is a controlled experiment that compares outcomes for two variants assigned to different users or experimental units.
AI Model Evaluation
A controlled online experiment that compares outcomes between two variants, typically a control and a treatment, using assigned users or traffic.
- How it is used
- Measures AI model changes against user, product, and business outcomes in production.
- How deep it goes
- Develops layered experiment design, readiness, model-specific risks, and interpretation of ambiguous results.
- Examples
- Uses a contextual movie recommender test and a latency degradation experiment with control and treatment variants.
- What you can do after
- Evaluate a model change by designing an online experiment and interpreting its results against product goals and operational trade-offs.
Hidden Influences
An online experiment that randomly exposes different user groups to alternative system variants and compares their outcomes.
- How it is used
- Uses live comparisons to decide whether to deploy a new recommender model.
- How deep it goes
- Explains random assignment, baseline comparison, historical screening, and the interpretation of competing metrics.
- Examples
- Uses a drug-versus-placebo trial, a model-comparison dashboard, and a social-media sentiment experiment.
- What you can do after
- Evaluate and understand how random assignment and metric choice affect comparisons between system variants.
Designing AI Systems
A controlled experiment that compares alternative variants by assigning them to different user or traffic groups and measuring outcomes.
- How it is used
- Validates prompt, model, or retrieval changes against real production traffic before promotion.
- How deep it goes
- Explains variants, allocation, assignment, metrics, and significance, but does not detail statistical test formulas.
- Examples
- Uses prompt comparisons measuring response quality, tool-selection accuracy, cost, latency, and sample size.
- What you can do after
- Operate the Experimentation Service to assign variants, record outcomes, compare metrics, and promote or reject production changes.
High Performance Unity Game Development
A/B testing is an experiment that compares two or more variants by assigning them to different user groups and measuring the resulting outcomes.
- How it is used
- Uses separate game-balance versions to test gameplay changes against outcomes such as player retention.
- How deep it goes
- Explains group assignment and outcome comparison through a reward example, without detailing statistical analysis.
- Examples
- Tests 50 versus 100 gold per level completion and describes running five balance variants.
- What you can do after
- Evaluate whether a balance change improves retention by comparing outcomes across user groups.
Read how High Performance Unity Game Development explains A/B test →
Spring Microservices in Action
A/B testing is a technique that presents different versions of a service or feature to different users to evaluate them before broader rollout.
- How it is used
- Evaluates a new service feature before broad rollout by directing only part of the user population to it.
- How deep it goes
- Works through service lookup, weighted random selection, and routing to an alternate endpoint.
- Examples
- Splits organization-service calls evenly between old and new versions in a 50/50 illustration.
- What you can do after
- Apply it via a Zuul route filter to direct a weighted share of calls to an alternate service version.
Also covered in
- Financial AI in Practice
- Prompt Engineering in Practice
- Vibe EngineeringIn early access (MEAP)
What to know first
- recommender systemA recommender system is a computational system that predicts, selects, or ranks items, content, or actions for users based on available information.
- metricA metric is a quantitative measure that assigns values to a property or outcome so it can be assessed, compared, or monitored.
- click-through rateClick-through rate is the proportion of impressions or opportunities to display an item that result in a click on that item.
- retentionRetention is a measure of the proportion or amount of a defined population, relationship, activity, or information that persists over a specified period.
Related concepts
- retentionmeasures · Retention is a measure of the proportion or amount of a defined population, relationship, activity, or information that persists over a specified period.
- offline evaluationcontrasts with · Offline evaluation is the assessment of a system using prepared or previously collected data outside live interactions or production traffic.
- baselineuses · A reference condition or measurement representing a neutral or expected state against which system outcomes are compared.
Questions about A/B test
What is A/B test?
An A/B test is a controlled experiment that compares outcomes for two variants assigned to different users or experimental units.
Which Manning books explain A/B test?
AI Model Evaluation, Hidden Influences, Designing AI Systems, High Performance Unity Game Development and Spring Microservices in Action.
What should I know before learning A/B test?
Manning books that teach A/B test usually introduce recommender system, metric, click-through rate and retention first.
What is the difference between A/B test and offline evaluation?
An A/B test is a controlled experiment that compares outcomes for two variants assigned to different users or experimental units. Offline evaluation is the assessment of a system using prepared or previously collected data outside live interactions or production traffic.
What is A/B test also called?
A/B test is also called A/B testing.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




