liveBook · Manning

Rank-aware metric

As explained in AI Model Evaluation

A ranking metric measures the quality of an ordered list of items relative to desired relevance or preference.

A rank-aware metric evaluates whether relevant items appear in an ordered list, usually giving special attention to the highest positions. It supports offline decisions about recommendation and search quality before exposing model changes to users. section 1.3.5 section 1.9

Choose the metric according to the product behavior being optimized:

  • Precision@K measures how many of the top K recommendations are relevant. section 1.3.5
  • Recall@K measures how many relevant items appear within the top K. section 1.3.5
  • NDCG@K measures whether highly relevant items occupy higher-ranked positions; MAP is another applicable ranking metric. section 1.3.5
  • Pair ranking scores with coverage, diversity, or novelty when repeated or overly narrow recommendations would damage the user experience. section 1.3.5

Movie recommendations

Suppose five shows are recommended and three are relevant according to watch history or labeled preference data. Precision@5 is 3/5, or 60%. That score does not establish that the list is diverse, novel, or satisfactory as a complete product experience. section 1.3.5

Table 1.2 Example of Offline Metrics for a Recommender Model
Metric
Metric Category
Definition
Precision at K (P@K)
Ranking and Retrieval Metric
Of the top K recommended items, how many are relevant? Useful when the quality of the top results matters most.
Recall at K (R@K)
Ranking and Retrieval Metric
Of all relevant items, how many appear in the top K? Useful when the system should avoid missing relevant options.
NDCG@K
Ranking Metric
Whether highly relevant items appear higher in the ranked list. Useful when position matters.
Coverage
Recommender System Metric
How much of the catalog or user base the recommender meaningfully serves. Useful for detecting over-concentration.
Diversity
Recommender System Metric
Whether the recommendation set contains varied items rather than many similar ones. Useful for avoiding repetitive experiences.
Groundedness
Generative AI / LLM Metric
Whether a generated recommendation explanation is supported by user history, item metadata, or retrieved context.

Compare ranking and retrieval measures with coverage, diversity, groundedness, and other metrics for different recommender-system components. section 1.3.5

Treat metric selection as a product decision rather than a purely technical choice. Run the metric on historical or freshly collected production data, then compare offline results with online outcomes; metric scores alone do not capture every user or system constraint. section 1.3.5 section 1.9

How AI Model Evaluation covers it

How deep it goes
Defines Precision@K, Recall@K, NDCG@K, and MAP conceptually, but does not derive their formulas beyond a Precision@5 example.
Examples
Uses a five-show recommendation list with three relevant titles to demonstrate Precision@5 at 60%.
What it assumes
Assumes labeled relevance or preference data and introduces offline evaluation as a way to test models before live exposure.
What you can do after
Evaluate recommendation changes with a position-sensitive offline metric and select complementary measures for broader product goals.
How it connects
Connects ranking metrics with retrieval, coverage, diversity, novelty, online experiments, and offline evaluation.

Also explained in

Compare how 2 books explain rank-aware metric →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.