Diversity measures how varied a set of recommendations is, rather than how many similar items it contains. It helps detect a stale experience: a model can rank relevant titles highly while repeatedly recommending a narrow selection. section 1.3.5 section 3.3.2
Choose measures that reflect the product goal. Intra-list diversity describes how different items in a recommendation list are from one another. Catalog coverage is a related measure of how much of the item pool is exposed; it is not the same as variety within one list. Attributes such as genre, content source, and popularity can inform the assessment. section 3.3.2
Use diversity as a diagnostic, not a goal to maximize blindly:
- Measure a baseline across recommendations and choose a minimum threshold that avoids overly narrow results. section 3.3.2
- Test the threshold offline, then refine it using user feedback and A/B testing. Compare it with engagement metrics such as click-through rate to identify trade-offs. section 3.3.2
- Monitor the diversity index in live logs. Keep category mappings, filters, aggregation windows, and formulas consistent between offline and production calculations, and alert when known risks appear. section 7.8.1
Genre variety in recommendations
For a top-10 recommendation list, a team might require at least three distinct genres. That threshold sets a minimum variety requirement while the team continues to assess engagement; it should be tested and adjusted rather than treated as universally correct. section 3.3.2
An LLM judge can assess variety differently: given viewing history and five recommended movies, it can score diversity from 1 to 5 based on genre, year, or theme, with a short justification. This qualitative judgment is distinct from a calculated diversity index. Check its stability, agreement with other models, and correlation with human ratings. section 9.6.1
Variety also matters in evaluation data. Sampling across user tenure, geography, or use cases can help an LLM judge avoid evaluating only a narrow distribution of easy examples. section 9.4.2
Table 1.2 Example of Offline Metrics for a Recommender Model
|
Metric
|
Metric Category
|
Definition
|
|
Precision at K (P@K)
|
Ranking and Retrieval Metric
|
Of the top K recommended items, how many are relevant? Useful when the quality of the top results matters most.
|
|
Recall at K (R@K)
|
Ranking and Retrieval Metric
|
Of all relevant items, how many appear in the top K? Useful when the system should avoid missing relevant options.
|
|
NDCG@K
|
Ranking Metric
|
Whether highly relevant items appear higher in the ranked list. Useful when position matters.
|
|
Coverage
|
Recommender System Metric
|
How much of the catalog or user base the recommender meaningfully serves. Useful for detecting over-concentration.
|
|
Diversity
|
Recommender System Metric
|
Whether the recommendation set contains varied items rather than many similar ones. Useful for avoiding repetitive experiences.
|
|
Groundedness
|
Generative AI / LLM Metric
|
Whether a generated recommendation explanation is supported by user history, item metadata, or retrieved context.
|
Compare diversity with ranking, coverage, and other recommender evaluation metrics. section 1.3.5









