Online-offline correlation measures how changes in offline evaluation metrics relate to outcomes in live experiments. Strong correlation helps teams prioritize promising candidates and avoid costly A/B tests that offline signals already discount. section 1.7.2
Reliable correlation depends on consistent instrumentation. Capture the signals used at inference time, model outputs, confidence scores, and downstream user behavior, then compare online outcomes with offline metrics computed on similar data. section 1.7.2
Build a longitudinal correlation practice by:
- Maintaining a rolling dataset that links each model or variant to its offline scores and online experiment outcomes. section 7.6.3
- Recording experiment dates, traffic allocation, duration, product domain, and model maturity alongside the measurements. section 7.6.3
- Preserving stable metric definitions and logging schemas so results remain comparable over time. section 7.6.3
- Comparing offline measures such as precision@K, recall@K, NDCG, diversity, fairness gap, and latency with online changes in CTR, conversion, retention, or engagement. section 7.6.3
Testing a metric assumption
A team may believe precision@10 predicts click-through rate. Each experiment adds the offline precision@10 score and the resulting CTR delta to the rolling dataset, allowing the team to test that assumption across multiple experiments rather than treating it as fact. section 7.6.3
Figure 7.2 illustrates the offline to online correlation feedback loop.
Use the feedback-loop figure to connect offline scoring, live experimentation, observed outcomes, and subsequent evaluation practice. section 7.6.3
