Novelty measures whether recommendations feel unfamiliar, non-obvious, or meaningfully different from what users have already seen. It matters because a system can earn strong clicks and watch time while gradually creating a repetitive, stale experience. section 8.4.3 section 1.3.5
Treat novelty as a product-quality signal rather than a replacement for relevance. A recommendation can be surprising but unwanted, so novelty is often interpreted alongside satisfaction or relevance when assessing whether discovery is working. section 8.4.3 section 1.3.5
Useful novelty-related measurements include:
- Intra-list diversity checks whether one session's recommendations vary rather than repeat similar items. section 8.4.3 section 1.3.5
- Catalog coverage tracks how much of the available catalog receives exposure over a time window, revealing over-concentration. section 8.4.3 section 1.3.5
- Serendipity attempts to capture recommendations that are both unexpected and satisfying, although it is noisier to measure. section 8.4.3
Streaming recommendation example
A model can achieve strong Precision@K while repeatedly recommending the same narrow set of TV shows or movies. Add novelty, diversity, or coverage to the offline evaluation when the product goal includes avoiding an over-personalized homepage. section 1.3.5
Table 1.2 Example of Offline Metrics for a Recommender Model
|
Metric
|
Metric Category
|
Definition
|
|
Precision at K (P@K)
|
Ranking and Retrieval Metric
|
Of the top K recommended items, how many are relevant? Useful when the quality of the top results matters most.
|
|
Recall at K (R@K)
|
Ranking and Retrieval Metric
|
Of all relevant items, how many appear in the top K? Useful when the system should avoid missing relevant options.
|
|
NDCG@K
|
Ranking Metric
|
Whether highly relevant items appear higher in the ranked list. Useful when position matters.
|
|
Coverage
|
Recommender System Metric
|
How much of the catalog or user base the recommender meaningfully serves. Useful for detecting over-concentration.
|
|
Diversity
|
Recommender System Metric
|
Whether the recommendation set contains varied items rather than many similar ones. Useful for avoiding repetitive experiences.
|
|
Groundedness
|
Generative AI / LLM Metric
|
Whether a generated recommendation explanation is supported by user history, item metadata, or retrieved context.
|
Compare novelty-related recommender metrics with ranking and retrieval measures when selecting an offline evaluation set. section 1.3.5
