liveBook · Manning

Covariate

As explained in AI Model Evaluation

A covariate is a measured variable associated with an outcome that can be included in statistical adjustment of an effect estimate.

A covariate is a measured variable associated with an outcome that helps explain predictable variation during statistical adjustment. In online experiments, pre-experiment user behavior can reduce noise without changing the treatment effect estimate. section 8.6.2

Use a covariate carefully:

  • Freeze its value at randomization. A post-treatment measure can contain effects caused by the experiment and remove part of the treatment impact. section 8.6.3
  • Check predictive strength before implementation. A covariate explaining little outcome variance may add complexity without materially improving precision. section 8.6.3
  • Inspect fairness implications. Historical behavior can encode unequal access across user groups and make an apparent improvement conceal a real gap. section 8.6.3

Podcast recommendation experiment

Weekly listening time varies widely across users, obscuring an eight-minute treatment lift. Four weeks of pre-experiment listening history correlate with later behavior, so CUPED removes predictable variation: the lift stays the same while residual variance falls and the confidence interval narrows. section 8.6.2

Figure 8.3 illustrates variance reduction with CUPED (Movie Recommender Example). In this experiment, the unadjusted analysis shows noisy control and treatment averages with wide error bars, making the small lift hard to detect. After applying CUPED using pre-experiment watch-time as a covariate, the treatment and control means remain essentially the same, but the error bars (confidence intervals) shrink. That’s the point: CUPED doesn’t change the treatment effect estimate (+2 minutes watch time), it reduces variance, allowing the same signal to be detected with fewer users or shorter tests.

Compare unadjusted and covariate-adjusted experiment results: treatment and control means remain nearly unchanged, while confidence intervals shrink around the same estimated lift. section 8.6.2

How AI Model Evaluation covers it

How deep it goes
Explains CUPED adjustment, timing rules, predictive-strength checks, leakage, bias amplification, and overfitting risks.
Examples
Uses podcast listening history, movie recommender watch-time, confidence intervals, and treatment-effect estimates as concrete examples.
What it assumes
Assumes randomized treatment assignment and outcome history, while teaching how to select and freeze an appropriate pre-experiment measure.
What you can do after
Apply CUPED or regression adjustment while preventing post-treatment leakage and checking predictive strength and fairness risks.
How it connects
Connects covariates to CUPED, regression adjustment, variance reduction, treatment effects, and randomized experiments.

Also explained in

Compare how 3 books explain covariate →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.