liveBook · Manning

Diversity

As explained in AI Model Evaluation

A diversity index is a quantitative measure summarizing the variety of items, categories, or outcomes in a set.

Diversity measures how varied a set of recommendations is, rather than how many similar items it contains. It helps detect a stale experience: a model can rank relevant titles highly while repeatedly recommending a narrow selection. section 1.3.5 section 3.3.2

Choose measures that reflect the product goal. Intra-list diversity describes how different items in a recommendation list are from one another. Catalog coverage is a related measure of how much of the item pool is exposed; it is not the same as variety within one list. Attributes such as genre, content source, and popularity can inform the assessment. section 3.3.2

Use diversity as a diagnostic, not a goal to maximize blindly:

  • Measure a baseline across recommendations and choose a minimum threshold that avoids overly narrow results. section 3.3.2
  • Test the threshold offline, then refine it using user feedback and A/B testing. Compare it with engagement metrics such as click-through rate to identify trade-offs. section 3.3.2
  • Monitor the diversity index in live logs. Keep category mappings, filters, aggregation windows, and formulas consistent between offline and production calculations, and alert when known risks appear. section 7.8.1

Genre variety in recommendations

For a top-10 recommendation list, a team might require at least three distinct genres. That threshold sets a minimum variety requirement while the team continues to assess engagement; it should be tested and adjusted rather than treated as universally correct. section 3.3.2

An LLM judge can assess variety differently: given viewing history and five recommended movies, it can score diversity from 1 to 5 based on genre, year, or theme, with a short justification. This qualitative judgment is distinct from a calculated diversity index. Check its stability, agreement with other models, and correlation with human ratings. section 9.6.1

Variety also matters in evaluation data. Sampling across user tenure, geography, or use cases can help an LLM judge avoid evaluating only a narrow distribution of easy examples. section 9.4.2

Table 1.2 Example of Offline Metrics for a Recommender Model
Metric
Metric Category
Definition
Precision at K (P@K)
Ranking and Retrieval Metric
Of the top K recommended items, how many are relevant? Useful when the quality of the top results matters most.
Recall at K (R@K)
Ranking and Retrieval Metric
Of all relevant items, how many appear in the top K? Useful when the system should avoid missing relevant options.
NDCG@K
Ranking Metric
Whether highly relevant items appear higher in the ranked list. Useful when position matters.
Coverage
Recommender System Metric
How much of the catalog or user base the recommender meaningfully serves. Useful for detecting over-concentration.
Diversity
Recommender System Metric
Whether the recommendation set contains varied items rather than many similar ones. Useful for avoiding repetitive experiences.
Groundedness
Generative AI / LLM Metric
Whether a generated recommendation explanation is supported by user history, item metadata, or retrieved context.

Compare diversity with ranking, coverage, and other recommender evaluation metrics. section 1.3.5

How AI Model Evaluation covers it

How deep it goes
Defines intra-list diversity and coverage, then gives threshold-setting and monitoring steps without specifying an index formula.
Examples
Uses a three-genre minimum for top-10 recommendations and a 1–5 LLM rating of top-five movies.
What you can do after
Evaluate recommendation variety by establishing a baseline, setting a minimum threshold, and checking it against engagement and user feedback.
How it connects
Links diversity to catalog coverage, click-through rate, A/B testing, and LLM-as-a-judge evaluation.

Also explained in

Compare how 10 books explain diversity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.