liveBook · Manning

Sample size

As explained in AI Model Evaluation

Sample size is the number of observations or participants included in a statistical experiment.

Sample size is the number of users or observations included in an experiment. In an online A/B test, it is an operational decision alongside duration, traffic allocation, exposure rules, and ramp-up. The chosen size determines how much exposure each variant receives before a product decision is made. section 6.3.1

Testing more model variants increases the required sample size or duration because each treatment needs enough exposure for statistically rigorous comparisons. Larger experiments also increase false-positive risk through multiple comparisons, so variants should be tied to clear hypotheses and interpreted with appropriate corrections. section 6.5.1

Use sample size as part of a complete online evaluation plan:

  • Define which users qualify through targeting criteria such as device type, region, or user tenure. section 6.3.1
  • Reserve enough traffic or duration for every treatment, rather than adding variants without a decision hypothesis. section 6.5.1
  • Use offline evaluation to narrow the variants that reach production, conserving A/B testing capacity while still measuring real user impact online. section 1.6
Figure 1.7 Leveraging offline evaluations to inform online experimentation strategy results in considerable optimizations. By reducing the number of model variants that graduate to the online experimentation stage, you're reducing the sample size for the A/B test, freeing up testing capacity for other A/B tests to run on the product and being more strategic with the changes you're exposing users to.

Shows how filtering variants before online experimentation can reduce the sample size required for the A/B test and free product testing capacity. section 1.6

How AI Model Evaluation covers it

How deep it goes
Defines sample size operationally and connects it to traffic, duration, power, and false-positive risk without giving a calculation method.
Examples
Uses recommendation variants emphasizing recency, user affinity, or diversity to illustrate why additional treatments require more exposure.
What it assumes
Assumes familiarity with online controlled experiments and distinguishes production evaluation from offline evaluation.
What you can do after
Operate/configure A/B tests by allocating exposure, limiting variants, and planning enough traffic or duration for trustworthy comparisons.
How it connects
Connects sample size to A/B testing, traffic allocation, experiment duration, model variants, statistical power, and multiple comparisons.

Also explained in

Compare how 5 books explain sample size →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.