Variance reduction decreases outcome noise so online experiments can detect small, real effects without changing the treatment effect being measured. It matters when user behavior is heterogeneous, effect sizes are modest, and raw averages are underpowered. section 8.6 section 8.8
Apply variance reduction by:
- Selecting pre-treatment behavior or user characteristics that explain outcome variation. section 8.6
- Freezing the covariate before randomization so treatment cannot influence the adjustment. section 8.6.2
- Using CUPED or regression adjustment to shrink residual variance and tighten confidence intervals. section 8.6.2 section 8.8
- Checking subgroup effects and aligning adjustments with the population the product strategy actually serves. section 8.6.1
Movie recommendation experiment
A recommender produces an 8-minute watch-time lift, but heavy users create wide variability. Applying CUPED with four weeks of frozen pre-experiment watch time leaves the lift essentially unchanged while reducing residual variance by roughly 50%; the confidence interval tightens and the result reaches significance at p=0.02. section 8.6.2
Figure 8.2 illustrates how raw averages can flatten heterogeneous effects. In this movie recommender example, heavy users show a clear lift in watch time under the new model, while casual users see little to no change. When averaged across all users, the treatment effect looks much smaller than it does for the subgroup that benefited most. This illustrates why top-line A/B results can mislead in AI evaluations without subgroup analysis or variance reduction.
Compare heavy and casual movie-streaming users to see how raw averages can flatten heterogeneous treatment effects. section 8.6.1
Figure 8.3 illustrates variance reduction with CUPED (Movie Recommender Example). In this experiment, the unadjusted analysis shows noisy control and treatment averages with wide error bars, making the small lift hard to detect. After applying CUPED using pre-experiment watch-time as a covariate, the treatment and control means remain essentially the same, but the error bars (confidence intervals) shrink. That’s the point: CUPED doesn’t change the treatment effect estimate (+2 minutes watch time), it reduces variance, allowing the same signal to be detected with fewer users or shorter tests.
Inspect how CUPED narrows confidence intervals while preserving the estimated treatment effect in a movie recommender experiment. section 8.6.2
