liveBook · Manning

Reciprocal rank

As explained in AI Recommender Systems

Reciprocal rank is one divided by the position of the first relevant item in a ranked result list, or zero when no relevant item appears.

Reciprocal rank measures how quickly the first relevant item appears in a ranked list: 1 divided by its position, or 0 when no relevant item appears. section 4.6.3

Position determines the score

A relevant item at position 1 gives a score of 1.0; at position 3, 0.33; with no relevant item, the score is 0. section 4.6.3

Apply reciprocal rank within mean reciprocal rank (MRR):

  • Compute each user's reciprocal rank, then average those scores across users. section 4.6.3
  • Use it when users typically want one good result, such as search, question answering, or play-next recommendations. section 4.6.3
Listing 4.8 Calculate metrics for different K values
 
for k in k_values: #A
  if k <= len(top_k_indices): #B
    top_k_rel = relevance_scores[:k] #C
    
    if np.sum(top_k_rel) > 0:  #D
      ideal_rel = np.sort(top_k_rel)[::-1] #E
      ndcg_k = ndcg_score([ideal_rel], [top_k_rel])
      all_metrics[k]['ndcg'].append(ndcg_k)
              
    precision_k = np.sum(top_k_rel) / k #F
    all_metrics[k]['precision'].append(precision_k)
              
    if len(test_indices) > 0:
      recall_k = np.sum(top_k_rel) / len(test_indices) #G
      all_metrics[k]['recall'].append(recall_k)
              
    first_relevant = np.where(top_k_rel == 1)[0] #H
    if len(first_relevant) > 0:
      mrr = 1.0 / (first_relevant[0] + 1)
      all_metrics[k]['mrr'].append(mrr)
    else:
      all_metrics[k]['mrr'].append(0.0)
              
    if np.sum(top_k_rel) > 0: #I
      ap = 0.0
      relevant_found = 0
      for i, rel in enumerate(top_k_rel):
        if rel == 1:
        relevant_found += 1
    ap += relevant_found / (i + 1)
    ap /= np.sum(top_k_rel)
     all_metrics[k]['map'].append(ap)
   else:
     all_metrics[k]['map'].append(0.0)

The metric calculation checks for the first relevant item and assigns the reciprocal of its one-based position, otherwise assigning zero. section 4.6.3

How AI Recommender Systems covers it

How deep it goes
Defines the position-based formula, gives numeric cases, and shows implementation logic without developing statistical limitations.
Examples
Uses first-relevant positions of 1 and 3, a no-match case, and a Python-style MRR calculation.
What it assumes
Assumes ranked results and binary relevance labels, while defining the reciprocal-rank calculation directly.
What you can do after
Evaluate/understand it by calculating the first relevant item's reciprocal position and averaging scores across users.
How it connects
Connects to mean reciprocal rank, relevance, ranked lists, precision, recall, and NDCG.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Recommender Systems. The section links point to the parts of the book it draws on.