liveBook · Manning

Ideal discounted cumulative gain

As explained in AI Recommender Systems

Ideal discounted cumulative gain is the maximum discounted cumulative gain obtainable when all relevant items occupy the best possible positions.

Ideal discounted cumulative gain (IDCG) is the highest discounted cumulative gain possible for a user: place every relevant item in the best available positions and order them ideally. It provides the denominator for normalized discounted cumulative gain (nDCG@k), making ranking scores comparable across users with different numbers of relevant items. section 4.6.3

Use IDCG to interpret normalized ranking evaluation:

  • Compute the actual DCG for the recommendation list at cutoff k. section 4.6.3
  • Compute IDCG@k by arranging the observed relevance scores in descending order, representing the best possible ranking. section 4.6.3
  • Divide DCG@k by IDCG@k; a perfect ranking produces nDCG = 1, while no relevant items produces nDCG = 0. section 4.6.3
Listing 4.8 Calculate metrics for different K values
 
for k in k_values: #A
  if k <= len(top_k_indices): #B
    top_k_rel = relevance_scores[:k] #C
    
    if np.sum(top_k_rel) > 0:  #D
      ideal_rel = np.sort(top_k_rel)[::-1] #E
      ndcg_k = ndcg_score([ideal_rel], [top_k_rel])
      all_metrics[k]['ndcg'].append(ndcg_k)
              
    precision_k = np.sum(top_k_rel) / k #F
    all_metrics[k]['precision'].append(precision_k)
              
    if len(test_indices) > 0:
      recall_k = np.sum(top_k_rel) / len(test_indices) #G
      all_metrics[k]['recall'].append(recall_k)
              
    first_relevant = np.where(top_k_rel == 1)[0] #H
    if len(first_relevant) > 0:
      mrr = 1.0 / (first_relevant[0] + 1)
      all_metrics[k]['mrr'].append(mrr)
    else:
      all_metrics[k]['mrr'].append(0.0)
              
    if np.sum(top_k_rel) > 0: #I
      ap = 0.0
      relevant_found = 0
      for i, rel in enumerate(top_k_rel):
        if rel == 1:
        relevant_found += 1
    ap += relevant_found / (i + 1)
    ap /= np.sum(top_k_rel)
     all_metrics[k]['map'].append(ap)
   else:
     all_metrics[k]['map'].append(0.0)

The metric-evaluation listing sorts top-k relevance scores into an ideal order before calculating nDCG. section 4.6.3

How AI Recommender Systems covers it

How deep it goes
Defines the ideal ranking and its role in nDCG, then demonstrates relevance sorting without deriving deeper implementation internals.
Examples
Uses a perfect ranking, an empty relevant set, and descending top-k relevance scores as evaluation examples.
What it assumes
Assumes relevance scores, ranked positions, discounted gain, and DCG before introducing ideal ordering and normalization.
What you can do after
Evaluate ranking quality by constructing an ideal relevance order and normalizing DCG at a chosen cutoff.
How it connects
Connects discounted gain and DCG to nDCG, while contrasting normalized evaluation with unnormalized DCG.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Recommender Systems. The section links point to the parts of the book it draws on.