liveBook · Manning

Beyond-accuracy metric

As explained in AI Recommender Systems

A beyond-accuracy metric evaluates recommendation qualities such as coverage, diversity, or serendipity that are not captured by relevance accuracy alone.

A beyond-accuracy metric evaluates recommendation qualities that relevance and ranking metrics cannot capture. It asks whether recommendations reach different users, cover the catalog, vary within a list, or surface unexpected items. section 4.1.3 section 4.6

Use beyond-accuracy metrics to expose weaknesses that strong precision or ranking scores can hide:

  • User coverage measures performance across user segments, revealing whether cold-start users receive useful personalization or generic fallbacks. section 4.6.4
  • Item coverage measures the fraction of the catalog that appears in anyone's recommendations, exposing popularity bias and long-tail invisibility. section 4.6.4
  • Diversity measures variety within one user's list; recommending different facets of an interest can be more useful than repeating highly similar items. section 4.6.4
  • Serendipity captures surprising recommendations that accuracy-focused systems may never surface. section 4.6 section 4.6.4

Diagnosing catalog concentration

An ALS model recommends only 2.6% of the catalog, with more than half of those items from the most popular quartile. Comparing this result with a popularity baseline helps identify whether personalization is narrowing exposure rather than expanding it. section 4.6.4

Figure 4.18 ALS recommends only a small percentage of available items.

Inspect the catalog-coverage visualization to see how a recommender can concentrate exposure in a small portion of available items. section 4.6.4

How AI Recommender Systems covers it

How deep it goes
Defines coverage and diversity with production-oriented examples, while treating serendipity as a named dimension.
Examples
Uses user-activity hit rates, 2.6% item coverage, popularity baselines, and repeated action-movie recommendations.
What it assumes
Distinguishes relevance and ranking metrics before adding coverage, diversity, and serendipity.
What you can do after
Evaluate recommendation systems for user reach, catalog exposure, within-list variety, and surprising results.
How it connects
Connects accuracy metrics, ranking metrics, cold start, popularity bias, precision, recall, and NDCG.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Recommender Systems. The section links point to the parts of the book it draws on.