A covariate is a measured variable associated with an outcome that helps explain predictable variation during statistical adjustment. In online experiments, pre-experiment user behavior can reduce noise without changing the treatment effect estimate. section 8.6.2
Use a covariate carefully:
- Freeze its value at randomization. A post-treatment measure can contain effects caused by the experiment and remove part of the treatment impact. section 8.6.3
- Check predictive strength before implementation. A covariate explaining little outcome variance may add complexity without materially improving precision. section 8.6.3
- Inspect fairness implications. Historical behavior can encode unequal access across user groups and make an apparent improvement conceal a real gap. section 8.6.3
Podcast recommendation experiment
Weekly listening time varies widely across users, obscuring an eight-minute treatment lift. Four weeks of pre-experiment listening history correlate with later behavior, so CUPED removes predictable variation: the lift stays the same while residual variance falls and the confidence interval narrows. section 8.6.2
Figure 8.3 illustrates variance reduction with CUPED (Movie Recommender Example). In this experiment, the unadjusted analysis shows noisy control and treatment averages with wide error bars, making the small lift hard to detect. After applying CUPED using pre-experiment watch-time as a covariate, the treatment and control means remain essentially the same, but the error bars (confidence intervals) shrink. That’s the point: CUPED doesn’t change the treatment effect estimate (+2 minutes watch time), it reduces variance, allowing the same signal to be detected with fewer users or shorter tests.
Compare unadjusted and covariate-adjusted experiment results: treatment and control means remain nearly unchanged, while confidence intervals shrink around the same estimated lift. section 8.6.2


