liveBook · Manning

Variance reduction

As explained in AI Model Evaluation

Variance reduction is the use of statistical methods to decrease outcome variability and improve the precision or power of an estimate.

Variance reduction decreases outcome noise so online experiments can detect small, real effects without changing the treatment effect being measured. It matters when user behavior is heterogeneous, effect sizes are modest, and raw averages are underpowered. section 8.6 section 8.8

Apply variance reduction by:

  • Selecting pre-treatment behavior or user characteristics that explain outcome variation. section 8.6
  • Freezing the covariate before randomization so treatment cannot influence the adjustment. section 8.6.2
  • Using CUPED or regression adjustment to shrink residual variance and tighten confidence intervals. section 8.6.2 section 8.8
  • Checking subgroup effects and aligning adjustments with the population the product strategy actually serves. section 8.6.1

Movie recommendation experiment

A recommender produces an 8-minute watch-time lift, but heavy users create wide variability. Applying CUPED with four weeks of frozen pre-experiment watch time leaves the lift essentially unchanged while reducing residual variance by roughly 50%; the confidence interval tightens and the result reaches significance at p=0.02. section 8.6.2

Figure 8.2 illustrates how raw averages can flatten heterogeneous effects. In this movie recommender example, heavy users show a clear lift in watch time under the new model, while casual users see little to no change. When averaged across all users, the treatment effect looks much smaller than it does for the subgroup that benefited most. This illustrates why top-line A/B results can mislead in AI evaluations without subgroup analysis or variance reduction.

Compare heavy and casual movie-streaming users to see how raw averages can flatten heterogeneous treatment effects. section 8.6.1

Figure 8.3 illustrates variance reduction with CUPED (Movie Recommender Example). In this experiment, the unadjusted analysis shows noisy control and treatment averages with wide error bars, making the small lift hard to detect. After applying CUPED using pre-experiment watch-time as a covariate, the treatment and control means remain essentially the same, but the error bars (confidence intervals) shrink. That’s the point: CUPED doesn’t change the treatment effect estimate (+2 minutes watch time), it reduces variance, allowing the same signal to be detected with fewer users or shorter tests.

Inspect how CUPED narrows confidence intervals while preserving the estimated treatment effect in a movie recommender experiment. section 8.6.2

How AI Model Evaluation covers it

How deep it goes
Explains covariate selection, timing, CUPED, regression adjustment, and subgroup alignment, but does not derive their statistical formulas.
Examples
Uses movie recommendation watch time, heavy-versus-casual users, pre-experiment history, confidence intervals, and an 8-minute lift.
What it assumes
Assumes online treatment-control experiments and baseline user behavior, while teaching why covariates must precede treatment assignment.
What you can do after
Diagnose noisy online experiments and apply CUPED or regression adjustment without contaminating treatment-effect estimates.
How it connects
Connects CUPED, regression adjustment, covariate adjustment, A/B testing, subgroup analysis, and heterogeneous treatment effects.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.