Sample size is the number of users or observations included in an experiment. In an online A/B test, it is an operational decision alongside duration, traffic allocation, exposure rules, and ramp-up. The chosen size determines how much exposure each variant receives before a product decision is made. section 6.3.1
Testing more model variants increases the required sample size or duration because each treatment needs enough exposure for statistically rigorous comparisons. Larger experiments also increase false-positive risk through multiple comparisons, so variants should be tied to clear hypotheses and interpreted with appropriate corrections. section 6.5.1
Use sample size as part of a complete online evaluation plan:
- Define which users qualify through targeting criteria such as device type, region, or user tenure. section 6.3.1
- Reserve enough traffic or duration for every treatment, rather than adding variants without a decision hypothesis. section 6.5.1
- Use offline evaluation to narrow the variants that reach production, conserving A/B testing capacity while still measuring real user impact online. section 1.6
Figure 1.7 Leveraging offline evaluations to inform online experimentation strategy results in considerable optimizations. By reducing the number of model variants that graduate to the online experimentation stage, you're reducing the sample size for the A/B test, freeing up testing capacity for other A/B tests to run on the product and being more strategic with the changes you're exposing users to.
Shows how filtering variants before online experimentation can reduce the sample size required for the A/B test and free product testing capacity. section 1.6




