liveBook · Manning

Control group

As explained in AI Model Evaluation

A control variant is the reference condition in an experiment against which treatment variants are compared.

A control group is the reference condition in an online experiment: it does not receive the change, while treatment groups do. Comparing outcomes between them estimates the change’s effect on real users. section 6.1

Choose the reference according to the change being tested. For a new model, use a heuristic; for an incremental production change, use the current production model. Keep assignment and exposure consistent so metric differences remain interpretable. section 6.3 section 6.3.6

Evaluate a control group alongside:

  • Primary user outcomes, such as recommendation click-through rate and average watch time. section 6.3.6
  • System and experience measures, such as page-load latency, bounce rate, abandonment, errors, and satisfaction signals. section 4.4.3 section 6.3.6
  • Confounders and operational conditions, including device type, caching, warm-up effects, sample size, and experiment overlap. section 4.4.3 section 6.3.6

Movie recommendations

Use the current production recommendation model as control and compare it with two versions adding contextual features. In a mobile latency test, the control adds 0 ms, loads in 1050 ms, reaches 23.5% CTR, and has a 27.8% bounce rate; added delays increase load time and worsen both user metrics. section 6.3.6 section 4.4.3

Figure 6.2 A/B testing design layers, organized from outer to inner circles. The closer to the core, the more foundational and irreversible the decisions tend to be: the outer execution layer covers operational factors like test duration, rollout timing, and sample size; the middle experimental design layer defines how the test is structured, including variants, metrics, and assignment logic; and the inner core layer captures the underlying purpose of the model and the user or product behavior it’s intended to influence.

Use the layered experiment design to connect the control group with assignment logic, metrics, rollout timing, sample size, and the intended user or product behavior. section 6.3

How AI Model Evaluation covers it

How deep it goes
Defines control selection and compares outcomes, while deferring deeper statistical analysis and confounder treatment.
Examples
Uses heuristic, production-model, contextual-feature, and latency-degradation controls in recommendation experiments.
What it assumes
Assumes familiarity with models and product metrics, while defining A/B testing and control-versus-treatment comparison.
What you can do after
Evaluate model changes against user outcomes and guardrails in a real product environment.
How it connects
Connects A/B testing with treatment groups, variants, randomization, metrics, guardrails, and online model evaluation.

Also explained in

Compare how 4 books explain control group →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.