CUPED reduces noise in randomized experiments by adjusting outcomes with pre-treatment behavior correlated with those outcomes. It is useful when heterogeneous users and small effects make raw averages hard to interpret. section 8.6 section 8.8
The method subtracts the predictable portion of each outcome using a covariate frozen before treatment assignment. It sharpens the estimate rather than changing the treatment effect. section 8.6.2
Apply CUPED with these safeguards:
- Collect the covariate before randomization and freeze it at assignment. section 8.6.2
- Prefer historical behavior that predicts the post-experiment outcome, such as four weeks of prior listening or watch time. section 8.6.2
- Check covariate choices carefully: poor choices can amplify bias, while adjustment does not correct segment disparities or flawed product metrics. section 8.8
Listening-time experiment
An experiment measures an 8-minute lift in weekly listening time, but user behavior varies widely and the raw result has a p-value of 0.09. Four weeks of pre-experiment listening correlate with post-experiment behavior at about 0.7. CUPED reduces residual variance by roughly 50%, keeps the lift unchanged, and tightens the result to a p-value of 0.02. section 8.6.2
Figure 8.3 illustrates variance reduction with CUPED (Movie Recommender Example). In this experiment, the unadjusted analysis shows noisy control and treatment averages with wide error bars, making the small lift hard to detect. After applying CUPED using pre-experiment watch-time as a covariate, the treatment and control means remain essentially the same, but the error bars (confidence intervals) shrink. That’s the point: CUPED doesn’t change the treatment effect estimate (+2 minutes watch time), it reduces variance, allowing the same signal to be detected with fewer users or shorter tests.
Use the figure to compare unadjusted and CUPED-adjusted movie recommender results: treatment and control means remain similar, while confidence intervals shrink around the same +2-minute effect. section 8.6.2
