liveBook · Manning

A/B test

Also called A/B testing

An A/B test is a controlled experiment that compares outcomes for two variants assigned to different users or experimental units.

How 5 Manning books explain it

AI Model Evaluation

A controlled online experiment that compares outcomes between two variants, typically a control and a treatment, using assigned users or traffic.

How it is used
Measures AI model changes against user, product, and business outcomes in production.
How deep it goes
Develops layered experiment design, readiness, model-specific risks, and interpretation of ambiguous results.
Examples
Uses a contextual movie recommender test and a latency degradation experiment with control and treatment variants.
What you can do after
Evaluate a model change by designing an online experiment and interpreting its results against product goals and operational trade-offs.

Read how AI Model Evaluation explains A/B test →

Hidden Influences

An online experiment that randomly exposes different user groups to alternative system variants and compares their outcomes.

How it is used
Uses live comparisons to decide whether to deploy a new recommender model.
How deep it goes
Explains random assignment, baseline comparison, historical screening, and the interpretation of competing metrics.
Examples
Uses a drug-versus-placebo trial, a model-comparison dashboard, and a social-media sentiment experiment.
What you can do after
Evaluate and understand how random assignment and metric choice affect comparisons between system variants.

Read how Hidden Influences explains A/B test →

Designing AI Systems

A controlled experiment that compares alternative variants by assigning them to different user or traffic groups and measuring outcomes.

How it is used
Validates prompt, model, or retrieval changes against real production traffic before promotion.
How deep it goes
Explains variants, allocation, assignment, metrics, and significance, but does not detail statistical test formulas.
Examples
Uses prompt comparisons measuring response quality, tool-selection accuracy, cost, latency, and sample size.
What you can do after
Operate the Experimentation Service to assign variants, record outcomes, compare metrics, and promote or reject production changes.

Read how Designing AI Systems explains A/B test →

High Performance Unity Game Development

A/B testing is an experiment that compares two or more variants by assigning them to different user groups and measuring the resulting outcomes.

How it is used
Uses separate game-balance versions to test gameplay changes against outcomes such as player retention.
How deep it goes
Explains group assignment and outcome comparison through a reward example, without detailing statistical analysis.
Examples
Tests 50 versus 100 gold per level completion and describes running five balance variants.
What you can do after
Evaluate whether a balance change improves retention by comparing outcomes across user groups.

Read how High Performance Unity Game Development explains A/B test →

Spring Microservices in Action

A/B testing is a technique that presents different versions of a service or feature to different users to evaluate them before broader rollout.

How it is used
Evaluates a new service feature before broad rollout by directing only part of the user population to it.
How deep it goes
Works through service lookup, weighted random selection, and routing to an alternate endpoint.
Examples
Splits organization-service calls evenly between old and new versions in a 50/50 illustration.
What you can do after
Apply it via a Zuul route filter to direct a weighted share of calls to an alternate service version.

Read how Spring Microservices in Action explains A/B test →

Also covered in

What to know first

Related concepts

Questions about A/B test

What is A/B test?

An A/B test is a controlled experiment that compares outcomes for two variants assigned to different users or experimental units.

Which Manning books explain A/B test?

AI Model Evaluation, Hidden Influences, Designing AI Systems, High Performance Unity Game Development and Spring Microservices in Action.

What should I know before learning A/B test?

Manning books that teach A/B test usually introduce recommender system, metric, click-through rate and retention first.

What is the difference between A/B test and offline evaluation?

An A/B test is a controlled experiment that compares outcomes for two variants assigned to different users or experimental units. Offline evaluation is the assessment of a system using prepared or previously collected data outside live interactions or production traffic.

What is A/B test also called?

A/B test is also called A/B testing.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.