A control group is the reference condition in an online experiment: it does not receive the change, while treatment groups do. Comparing outcomes between them estimates the change’s effect on real users. section 6.1
Choose the reference according to the change being tested. For a new model, use a heuristic; for an incremental production change, use the current production model. Keep assignment and exposure consistent so metric differences remain interpretable. section 6.3 section 6.3.6
Evaluate a control group alongside:
- Primary user outcomes, such as recommendation click-through rate and average watch time. section 6.3.6
- System and experience measures, such as page-load latency, bounce rate, abandonment, errors, and satisfaction signals. section 4.4.3 section 6.3.6
- Confounders and operational conditions, including device type, caching, warm-up effects, sample size, and experiment overlap. section 4.4.3 section 6.3.6
Movie recommendations
Use the current production recommendation model as control and compare it with two versions adding contextual features. In a mobile latency test, the control adds 0 ms, loads in 1050 ms, reaches 23.5% CTR, and has a 27.8% bounce rate; added delays increase load time and worsen both user metrics. section 6.3.6 section 4.4.3
Figure 6.2 A/B testing design layers, organized from outer to inner circles. The closer to the core, the more foundational and irreversible the decisions tend to be: the outer execution layer covers operational factors like test duration, rollout timing, and sample size; the middle experimental design layer defines how the test is structured, including variants, metrics, and assignment logic; and the inner core layer captures the underlying purpose of the model and the user or product behavior it’s intended to influence.
Use the layered experiment design to connect the control group with assignment logic, metrics, rollout timing, sample size, and the intended user or product behavior. section 6.3



