Normalized discounted cumulative gain (NDCG) measures the quality of a ranked recommendation list. Precision counts relevant items but ignores their positions; NDCG gives more credit when those items appear near the top. It is useful when users are more likely to notice the first few results. section 4.6.3 section 4.8
Calculate NDCG@k from three components:
- Assign each item a gain. A common starting point is 1 if the user consumed it and 0 otherwise; graded relevance is also possible. section 4.6.3
- Discount gains by position and sum them to get discounted cumulative gain (DCG). With the illustrated logarithmic discount, positions 1, 2, and 3 receive weights of 1.0, 0.63, and 0.5. section 4.6.3
- Divide DCG@k by ideal DCG@k, the score for the best possible ordering: NDCG@k = DCG@k / IDCG@k. A perfect ranking scores 1. section 4.6.3
One relevant item, two positions
Two lists each contain one relevant item, so both have the same cumulative gain. Put that item first in one list and seventh in the other: discounting gives the first list the higher score. Normalization then makes the resulting DCG scores more comparable across users with different numbers of relevant test items. section 4.6.3
For an offline comparison, compute NDCG at the list lengths that matter to the product and compare it with a popularity baseline and the current model. Inspect scores across users, not only the mean. A temporal train/test split better simulates deployment than treating historical interactions as timeless ground truth. section 4.10 section 4.8
Use NDCG to assess the scoring stage, where position matters. Use recall to check whether retrieval found relevant candidates at all; use coverage and diversity to examine qualities NDCG misses. A higher NDCG does not, by itself, settle whether a model is worth deploying or A/B testing. section 4.10 section 4.8
Listing 4.8 Calculate metrics for different K values
for k in k_values: #A
if k <= len(top_k_indices): #B
top_k_rel = relevance_scores[:k] #C
if np.sum(top_k_rel) > 0: #D
ideal_rel = np.sort(top_k_rel)[::-1] #E
ndcg_k = ndcg_score([ideal_rel], [top_k_rel])
all_metrics[k]['ndcg'].append(ndcg_k)
precision_k = np.sum(top_k_rel) / k #F
all_metrics[k]['precision'].append(precision_k)
if len(test_indices) > 0:
recall_k = np.sum(top_k_rel) / len(test_indices) #G
all_metrics[k]['recall'].append(recall_k)
first_relevant = np.where(top_k_rel == 1)[0] #H
if len(first_relevant) > 0:
mrr = 1.0 / (first_relevant[0] + 1)
all_metrics[k]['mrr'].append(mrr)
else:
all_metrics[k]['mrr'].append(0.0)
if np.sum(top_k_rel) > 0: #I
ap = 0.0
relevant_found = 0
for i, rel in enumerate(top_k_rel):
if rel == 1:
relevant_found += 1
ap += relevant_found / (i + 1)
ap /= np.sum(top_k_rel)
all_metrics[k]['map'].append(ap)
else:
all_metrics[k]['map'].append(0.0)The metric loop slices relevance scores at each k, sorts that slice to form its ideal ordering, and calls ndcg_score when the slice contains a relevant item. section 4.6.3




