liveBook · Manning

CUPED

As explained in AI Model Evaluation

CUPED is a variance-reduction technique for randomized experiments that adjusts outcomes using pre-treatment covariates correlated with those outcomes.

CUPED reduces noise in randomized experiments by adjusting outcomes with pre-treatment behavior correlated with those outcomes. It is useful when heterogeneous users and small effects make raw averages hard to interpret. section 8.6 section 8.8

The method subtracts the predictable portion of each outcome using a covariate frozen before treatment assignment. It sharpens the estimate rather than changing the treatment effect. section 8.6.2

Apply CUPED with these safeguards:

  • Collect the covariate before randomization and freeze it at assignment. section 8.6.2
  • Prefer historical behavior that predicts the post-experiment outcome, such as four weeks of prior listening or watch time. section 8.6.2
  • Check covariate choices carefully: poor choices can amplify bias, while adjustment does not correct segment disparities or flawed product metrics. section 8.8

Listening-time experiment

An experiment measures an 8-minute lift in weekly listening time, but user behavior varies widely and the raw result has a p-value of 0.09. Four weeks of pre-experiment listening correlate with post-experiment behavior at about 0.7. CUPED reduces residual variance by roughly 50%, keeps the lift unchanged, and tightens the result to a p-value of 0.02. section 8.6.2

Figure 8.3 illustrates variance reduction with CUPED (Movie Recommender Example). In this experiment, the unadjusted analysis shows noisy control and treatment averages with wide error bars, making the small lift hard to detect. After applying CUPED using pre-experiment watch-time as a covariate, the treatment and control means remain essentially the same, but the error bars (confidence intervals) shrink. That’s the point: CUPED doesn’t change the treatment effect estimate (+2 minutes watch time), it reduces variance, allowing the same signal to be detected with fewer users or shorter tests.

Use the figure to compare unadjusted and CUPED-adjusted movie recommender results: treatment and control means remain similar, while confidence intervals shrink around the same +2-minute effect. section 8.6.2

How AI Model Evaluation covers it

How deep it goes
Explains the adjustment rationale, numerical effect, and covariate timing rule, but does not develop the estimator mathematically.
Examples
Uses listening-time and movie-recommender experiments with pre-experiment behavior, confidence intervals, and small measured lifts.
What it assumes
Assumes randomized treatment assignment and pre-experiment behavioral data, while teaching why treatment-contaminated covariates are invalid.
What you can do after
Apply it via pre-treatment covariates to detect small online experiment effects with fewer users or shorter tests.
How it connects
Connects randomized experiments with variance reduction, pre-treatment covariates, regression adjustment, and heterogeneous treatment effects.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.