liveBook · Manning

Correlation

As explained in AI Model Evaluation

Correlation is a statistical association between variables that does not by itself establish that one variable causes another.

Correlation measures how strongly two variables move together. A value near +1 indicates that they rise and fall together, a value near −1 indicates opposite movement, and a value near 0 indicates little relationship. Correlation does not establish that one variable causes another. section 7.6.3 section 5.1

In model evaluation, correlation connects offline scores with online experiment outcomes. A useful process maintains a rolling dataset containing variant identifiers, experiment dates, offline metrics, online metric changes, traffic allocation, experiment length, product domain, and model maturity. section 7.6.3

Use the relationship to test whether offline improvements predict product results:

  • Record offline measures such as precision@K, recall@K, NDCG, diversity, fairness gaps, and latency. section 7.6.3
  • Record corresponding online changes such as click-through rate, conversion, retention, engagement depth, or watch time. section 7.6.3
  • Compare each offline metric with each online metric over repeated experiments rather than assuming that one offline score predicts success. section 7.6.3
  • Update the relationships over time because metric definitions, logging schemas, and product conditions can change. section 7.6.3 section 7.8.3

Movie recommendation evaluation

Suppose precision@10 improves for a recommendation variant. Correlation tracking asks whether experiments also show higher click-through rate, watch time, or retention. A positive relationship can support model-selection decisions, but it cannot prove that the recommendation caused the outcome; causal inference is needed for that question. section 7.6.3 section 5.1

Figure 7.2 illustrates the offline to online correlation feedback loop.

Trace the feedback loop linking offline evaluation results, online experiments, historical correlations, and later model-selection decisions. section 7.6.3

How AI Model Evaluation covers it

How deep it goes
Explains directional strength and a rolling measurement process, but does not establish causation or specify a complete statistical method.
Examples
Uses precision@10, recall, diversity, click-through rate, watch time, retention, movie recommendations, and repeated A/B tests.
What it assumes
Assumes offline metrics and online experiments, while distinguishing association from causal inference.
What you can do after
Diagnose whether offline metric improvements reliably align with online outcomes using a maintained historical dataset.
How it connects
Links offline evaluation, online experimentation, A/B testing, model selection, causal inference, and counterfactual evaluation.

Also explained in

Compare how 22 books explain correlation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.