liveBook · Manning

Online-offline correlation

As explained in AI Model Evaluation

Offline-to-online correlation is the analysis of how changes in offline evaluation metrics relate to outcomes observed in live experiments.

Online-offline correlation measures how changes in offline evaluation metrics relate to outcomes in live experiments. Strong correlation helps teams prioritize promising candidates and avoid costly A/B tests that offline signals already discount. section 1.7.2

Reliable correlation depends on consistent instrumentation. Capture the signals used at inference time, model outputs, confidence scores, and downstream user behavior, then compare online outcomes with offline metrics computed on similar data. section 1.7.2

Build a longitudinal correlation practice by:

  • Maintaining a rolling dataset that links each model or variant to its offline scores and online experiment outcomes. section 7.6.3
  • Recording experiment dates, traffic allocation, duration, product domain, and model maturity alongside the measurements. section 7.6.3
  • Preserving stable metric definitions and logging schemas so results remain comparable over time. section 7.6.3
  • Comparing offline measures such as precision@K, recall@K, NDCG, diversity, fairness gap, and latency with online changes in CTR, conversion, retention, or engagement. section 7.6.3

Testing a metric assumption

A team may believe precision@10 predicts click-through rate. Each experiment adds the offline precision@10 score and the resulting CTR delta to the rolling dataset, allowing the team to test that assumption across multiple experiments rather than treating it as fact. section 7.6.3

Figure 7.2 illustrates the offline to online correlation feedback loop.

Use the feedback-loop figure to connect offline scoring, live experimentation, observed outcomes, and subsequent evaluation practice. section 7.6.3

How AI Model Evaluation covers it

How deep it goes
Defines correlation, specifies instrumentation and rolling-data procedures, and stops short of prescribing a single correlation method.
Examples
Compares precision@10, recall, diversity, fairness gap, or latency with CTR, conversion, retention, and engagement changes.
What it assumes
Assumes familiarity with offline metrics, online experiments, and user-behavior measures while defining correlation from basic movement patterns.
What you can do after
Apply it by maintaining a rolling dataset that tests whether offline metric improvements predict online experiment outcomes.
How it connects
Connects offline evaluation with online experimentation, A/B testing, model evaluation, and user-behavior metrics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.