liveBook · Manning

Novelty

As explained in AI Model Evaluation

Novelty measures how unfamiliar or non-obvious recommended items are to users or relative to prior recommendations.

Novelty measures whether recommendations feel unfamiliar, non-obvious, or meaningfully different from what users have already seen. It matters because a system can earn strong clicks and watch time while gradually creating a repetitive, stale experience. section 8.4.3 section 1.3.5

Treat novelty as a product-quality signal rather than a replacement for relevance. A recommendation can be surprising but unwanted, so novelty is often interpreted alongside satisfaction or relevance when assessing whether discovery is working. section 8.4.3 section 1.3.5

Useful novelty-related measurements include:

  • Intra-list diversity checks whether one session's recommendations vary rather than repeat similar items. section 8.4.3 section 1.3.5
  • Catalog coverage tracks how much of the available catalog receives exposure over a time window, revealing over-concentration. section 8.4.3 section 1.3.5
  • Serendipity attempts to capture recommendations that are both unexpected and satisfying, although it is noisier to measure. section 8.4.3

Streaming recommendation example

A model can achieve strong Precision@K while repeatedly recommending the same narrow set of TV shows or movies. Add novelty, diversity, or coverage to the offline evaluation when the product goal includes avoiding an over-personalized homepage. section 1.3.5

Table 1.2 Example of Offline Metrics for a Recommender Model
Metric
Metric Category
Definition
Precision at K (P@K)
Ranking and Retrieval Metric
Of the top K recommended items, how many are relevant? Useful when the quality of the top results matters most.
Recall at K (R@K)
Ranking and Retrieval Metric
Of all relevant items, how many appear in the top K? Useful when the system should avoid missing relevant options.
NDCG@K
Ranking Metric
Whether highly relevant items appear higher in the ranked list. Useful when position matters.
Coverage
Recommender System Metric
How much of the catalog or user base the recommender meaningfully serves. Useful for detecting over-concentration.
Diversity
Recommender System Metric
Whether the recommendation set contains varied items rather than many similar ones. Useful for avoiding repetitive experiences.
Groundedness
Generative AI / LLM Metric
Whether a generated recommendation explanation is supported by user history, item metadata, or retrieved context.

Compare novelty-related recommender metrics with ranking and retrieval measures when selecting an offline evaluation set. section 1.3.5

How AI Model Evaluation covers it

How deep it goes
Defines novelty and names intra-list diversity, catalog coverage, and serendipity, but does not provide calculation procedures.
Examples
Uses streaming rows of TV shows or movies that may repeatedly surface the same narrow set of titles.
What it assumes
Assumes a recommendation setting, available catalog or exposure data, and an offline evaluation context.
What you can do after
Evaluate recommendation changes offline with novelty-related measures alongside relevance metrics.
How it connects
Connects novelty with relevance, diversity, catalog coverage, serendipity, Precision@K, CTR, and user experience.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.