Covariate adjustment improves treatment-effect precision by accounting for pre-treatment user characteristics or behavior that predict the outcome. It is useful when heterogeneous users and small model lifts make raw A/B averages too noisy. section 8.6 section 8.8
CUPED subtracts the outcome variation predictable from pre-experiment behavior; regression adjustment holds baseline covariates constant. Neither changes the treatment effect being estimated—they reduce residual variance and tighten confidence intervals. section 8.6.2 section 8.8
Podcast recommendation experiment
A two-week test shows an 8-minute listening lift, but high user variance produces a p-value of 0.09. Four weeks of pre-experiment listening data correlates strongly with later listening; CUPED reduces residual variance by roughly 50%, leaving the lift unchanged while moving the p-value to 0.02. section 8.6.2
Figure 8.3 illustrates variance reduction with CUPED (Movie Recommender Example). In this experiment, the unadjusted analysis shows noisy control and treatment averages with wide error bars, making the small lift hard to detect. After applying CUPED using pre-experiment watch-time as a covariate, the treatment and control means remain essentially the same, but the error bars (confidence intervals) shrink. That’s the point: CUPED doesn’t change the treatment effect estimate (+2 minutes watch time), it reduces variance, allowing the same signal to be detected with fewer users or shorter tests.
Visualizes a movie recommender experiment where CUPED keeps the approximately two-minute treatment lift unchanged while shrinking confidence intervals. section 8.6.2
Apply covariate adjustment carefully:
- Freeze covariate data at randomization; never use behavior influenced by treatment, or the adjustment can remove part of the measured effect. section 8.6.3
- Check predictive strength before implementation; weak covariates add complexity while producing little variance reduction. section 8.6.3
- Inspect subgroup effects and structural differences, because adjustment can mask fairness gaps or amplify bias. section 8.6.3

