liveBook · Manning

Rank-aware metric

As explained in AI Recommender Systems

A ranking metric evaluates recommendation quality while accounting for the positions of relevant items in an ordered list.

A rank-aware metric scores an ordered recommendation list by both whether relevant items appear and how near the top they appear. Precision counts relevant items but cannot distinguish a list with its only relevant item in position 1 from one with that item in position 7. A rank-aware metric should score the first list higher. section 4.6.3

Normalized Discounted Cumulative Gain (NDCG) makes that distinction in three steps. Assign each item a gain, commonly 1 if the user consumed it and 0 otherwise. Discount gains at lower positions, then sum them to obtain Discounted Cumulative Gain (DCG). Finally, divide by Ideal DCG—the score for the best possible ordering—to make scores more comparable across users with different numbers of relevant items: nDCG@k = DCG@k / IDCG@k. section 4.6.3

Same items, different order

Consider two lists containing exactly one relevant item. Both have a cumulative gain of 1. With position discounting, the list placing that item first receives more credit than the list placing it farther down. A common logarithmic discount gives positions 1, 2, and 3 weights of approximately 1.0, 0.63, and 0.5. section 4.6.3

Choose k to match the number of recommendations being evaluated. NDCG, Mean Reciprocal Rank (MRR), and Mean Average Precision (MAP) are rank-aware metrics; the shown MRR calculation uses the reciprocal of the first relevant item’s position, or zero when none appears in the top k. As k grows, ranking metrics may plateau after relevant items have surfaced. section 4.6.3 section 4.6

Use rank-aware scores alongside other measures, not as a complete definition of recommendation quality. In a multi-stage pipeline, NDCG helps evaluate scoring, while recall evaluates retrieval and coverage and diversity evaluate re-ranking. Compare against baselines and inspect scores across users: an average can conceal poor experiences for particular groups. section 4.10

How AI Recommender Systems covers it

How deep it goes
Works through binary gain, positional discounting, DCG normalization, and a top-k calculation; introduces MRR and MAP without equivalent worked derivations.
Examples
Compares lists with one relevant item at different positions and calculates NDCG and MRR over different top-k values.
What it assumes
Assumes an ordered recommendation list and relevance judgments, while building NDCG from binary gain before allowing graded relevance.
What you can do after
Apply it via top-k metric calculations to compare ranked recommendations and assess scoring-stage quality.
How it connects
Contrasts positional scoring with precision and recall, and connects it to retrieval, re-ranking, coverage, and diversity.

Also explained in

Compare how 2 books explain rank-aware metric →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Recommender Systems. The section links point to the parts of the book it draws on.