liveBook · Manning

Normalized discounted cumulative gain

As explained in AI Recommender Systems

NDCG, or normalized discounted cumulative gain, measures ranked retrieval quality by rewarding relevant items more heavily when they appear near the top.

Normalized discounted cumulative gain (NDCG) measures the quality of a ranked recommendation list. Precision counts relevant items but ignores their positions; NDCG gives more credit when those items appear near the top. It is useful when users are more likely to notice the first few results. section 4.6.3 section 4.8

Calculate NDCG@k from three components:

  • Assign each item a gain. A common starting point is 1 if the user consumed it and 0 otherwise; graded relevance is also possible. section 4.6.3
  • Discount gains by position and sum them to get discounted cumulative gain (DCG). With the illustrated logarithmic discount, positions 1, 2, and 3 receive weights of 1.0, 0.63, and 0.5. section 4.6.3
  • Divide DCG@k by ideal DCG@k, the score for the best possible ordering: NDCG@k = DCG@k / IDCG@k. A perfect ranking scores 1. section 4.6.3

One relevant item, two positions

Two lists each contain one relevant item, so both have the same cumulative gain. Put that item first in one list and seventh in the other: discounting gives the first list the higher score. Normalization then makes the resulting DCG scores more comparable across users with different numbers of relevant test items. section 4.6.3

For an offline comparison, compute NDCG at the list lengths that matter to the product and compare it with a popularity baseline and the current model. Inspect scores across users, not only the mean. A temporal train/test split better simulates deployment than treating historical interactions as timeless ground truth. section 4.10 section 4.8

Use NDCG to assess the scoring stage, where position matters. Use recall to check whether retrieval found relevant candidates at all; use coverage and diversity to examine qualities NDCG misses. A higher NDCG does not, by itself, settle whether a model is worth deploying or A/B testing. section 4.10 section 4.8

Listing 4.8 Calculate metrics for different K values
 
for k in k_values: #A
  if k <= len(top_k_indices): #B
    top_k_rel = relevance_scores[:k] #C
    
    if np.sum(top_k_rel) > 0:  #D
      ideal_rel = np.sort(top_k_rel)[::-1] #E
      ndcg_k = ndcg_score([ideal_rel], [top_k_rel])
      all_metrics[k]['ndcg'].append(ndcg_k)
              
    precision_k = np.sum(top_k_rel) / k #F
    all_metrics[k]['precision'].append(precision_k)
              
    if len(test_indices) > 0:
      recall_k = np.sum(top_k_rel) / len(test_indices) #G
      all_metrics[k]['recall'].append(recall_k)
              
    first_relevant = np.where(top_k_rel == 1)[0] #H
    if len(first_relevant) > 0:
      mrr = 1.0 / (first_relevant[0] + 1)
      all_metrics[k]['mrr'].append(mrr)
    else:
      all_metrics[k]['mrr'].append(0.0)
              
    if np.sum(top_k_rel) > 0: #I
      ap = 0.0
      relevant_found = 0
      for i, rel in enumerate(top_k_rel):
        if rel == 1:
        relevant_found += 1
    ap += relevant_found / (i + 1)
    ap /= np.sum(top_k_rel)
     all_metrics[k]['map'].append(ap)
   else:
     all_metrics[k]['map'].append(0.0)

The metric loop slices relevance scores at each k, sorts that slice to form its ideal ordering, and calls ndcg_score when the slice contains a relevant item. section 4.6.3

How AI Recommender Systems covers it

How deep it goes
Builds from gain through position discounting and DCG to normalization, then shows a metric-loop implementation.
Examples
Compares lists with one relevant item at different positions and calculates NDCG at multiple k values.
What it assumes
Starts with binary relevance and explains gain, discounting, and normalization before using relevance scores in code.
What you can do after
Apply it via ndcg_score and NumPy to compare ranked recommendations at selected list lengths.
How it connects
Contrasts NDCG with precision and retrieval recall, and pairs it with coverage and diversity for broader evaluation.

Also explained in

Compare how 5 books explain normalized discounted cumulative gain →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Recommender Systems. The section links point to the parts of the book it draws on.