A rank-aware metric evaluates whether relevant items appear in an ordered list, usually giving special attention to the highest positions. It supports offline decisions about recommendation and search quality before exposing model changes to users. section 1.3.5 section 1.9
Choose the metric according to the product behavior being optimized:
- Precision@K measures how many of the top K recommendations are relevant. section 1.3.5
- Recall@K measures how many relevant items appear within the top K. section 1.3.5
- NDCG@K measures whether highly relevant items occupy higher-ranked positions; MAP is another applicable ranking metric. section 1.3.5
- Pair ranking scores with coverage, diversity, or novelty when repeated or overly narrow recommendations would damage the user experience. section 1.3.5
Movie recommendations
Suppose five shows are recommended and three are relevant according to watch history or labeled preference data. Precision@5 is 3/5, or 60%. That score does not establish that the list is diverse, novel, or satisfactory as a complete product experience. section 1.3.5
Table 1.2 Example of Offline Metrics for a Recommender Model
|
Metric
|
Metric Category
|
Definition
|
|
Precision at K (P@K)
|
Ranking and Retrieval Metric
|
Of the top K recommended items, how many are relevant? Useful when the quality of the top results matters most.
|
|
Recall at K (R@K)
|
Ranking and Retrieval Metric
|
Of all relevant items, how many appear in the top K? Useful when the system should avoid missing relevant options.
|
|
NDCG@K
|
Ranking Metric
|
Whether highly relevant items appear higher in the ranked list. Useful when position matters.
|
|
Coverage
|
Recommender System Metric
|
How much of the catalog or user base the recommender meaningfully serves. Useful for detecting over-concentration.
|
|
Diversity
|
Recommender System Metric
|
Whether the recommendation set contains varied items rather than many similar ones. Useful for avoiding repetitive experiences.
|
|
Groundedness
|
Generative AI / LLM Metric
|
Whether a generated recommendation explanation is supported by user history, item metadata, or retrieved context.
|
Compare ranking and retrieval measures with coverage, diversity, groundedness, and other metrics for different recommender-system components. section 1.3.5
Treat metric selection as a product decision rather than a purely technical choice. Run the metric on historical or freshly collected production data, then compare offline results with online outcomes; metric scores alone do not capture every user or system constraint. section 1.3.5 section 1.9

