liveBook · Manning

Correlation matrix

As explained in AI Model Evaluation

A correlation matrix is a tabular array whose cells contain pairwise correlation values for variables represented by its rows and columns.

A correlation matrix organizes pairwise relationships between metrics in a table. Rows can represent offline metrics, columns online metrics, and each cell the strength of their relationship. section 7.6.3

Use the matrix to turn evaluation results into model-selection guidance:

  • Compare relationships such as precision@10 with online CTR lift or diversity index with online session length. section 7.6.3
  • Start with Pearson’s r when exact change magnitude matters, then check Spearman’s rho for robustness against outliers or non-linear patterns. section 7.6.3
  • Update the matrix after every experiment and increase the model-selection weight of metrics that consistently predict meaningful online outcomes. section 7.6.3
Figure 7.2 illustrates the offline to online correlation feedback loop.

Follow the offline-to-online correlation feedback loop from measured relationships to revised model-selection priorities. section 7.6.3

How AI Model Evaluation covers it

How deep it goes
Defines the table, gives example coefficients, and extends it into an experiment-by-experiment feedback loop without deriving correlation statistics.
Examples
Uses precision@10 with CTR lift and diversity index with session length as example metric pairs.
What it assumes
Assumes familiarity with offline metrics, online metrics, and correlation measures.
What you can do after
Evaluate offline metrics as predictors of online success and adjust their model-selection weight over time.
How it connects
Connects offline metrics, online metrics, Pearson’s r, Spearman’s rho, and model selection.

Also explained in

Compare how 6 books explain correlation matrix →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.