liveBook · Manning

Mean reciprocal rank

As explained in AI Recommender Systems

Mean reciprocal rank is the average reciprocal of the position of the first relevant result across users or queries.

In a “play next” recommendation, finding one good option quickly may matter more than finding every relevant option. Mean reciprocal rank (MRR) measures how early the first relevant item appears. For each user, divide 1 by that item's one-based position, then average those scores across users. If no relevant item appears, that user's score is 0. section 4.6.3

Three recommendation lists

Suppose the first relevant item appears at position 1 for one user and position 3 for another, while a third user sees none. Their reciprocal ranks are 1, about 0.33, and 0; MRR is about 0.44. Moving the second user's first relevant item higher raises the score. section 4.6.3

To calculate MRR at a chosen cutoff K, take each user's top-K relevance labels, find the first label equal to 1, and convert its zero-based index to a one-based rank before taking the reciprocal. Record 0 when there is no match. Changing K can change which users have a match; ranking metrics often plateau once relevant items have been surfaced. section 4.6.3 section 4.6

Listing 4.8 Calculate metrics for different K values
 
for k in k_values: #A
  if k <= len(top_k_indices): #B
    top_k_rel = relevance_scores[:k] #C
    
    if np.sum(top_k_rel) > 0:  #D
      ideal_rel = np.sort(top_k_rel)[::-1] #E
      ndcg_k = ndcg_score([ideal_rel], [top_k_rel])
      all_metrics[k]['ndcg'].append(ndcg_k)
              
    precision_k = np.sum(top_k_rel) / k #F
    all_metrics[k]['precision'].append(precision_k)
              
    if len(test_indices) > 0:
      recall_k = np.sum(top_k_rel) / len(test_indices) #G
      all_metrics[k]['recall'].append(recall_k)
              
    first_relevant = np.where(top_k_rel == 1)[0] #H
    if len(first_relevant) > 0:
      mrr = 1.0 / (first_relevant[0] + 1)
      all_metrics[k]['mrr'].append(mrr)
    else:
      all_metrics[k]['mrr'].append(0.0)
              
    if np.sum(top_k_rel) > 0: #I
      ap = 0.0
      relevant_found = 0
      for i, rel in enumerate(top_k_rel):
        if rel == 1:
        relevant_found += 1
    ap += relevant_found / (i + 1)
    ap /= np.sum(top_k_rel)
     all_metrics[k]['map'].append(ap)
   else:
     all_metrics[k]['map'].append(0.0)

The metric loop shows the first-relevant-item lookup, index adjustment, and zero-score branch alongside other top-K metrics. section 4.6.3

MRR answers a narrower question than precision or recall, which count relevant items, and MAP, which rewards placing multiple relevant items near the top. Evaluate it against relevance sets built from historical data, alongside baselines and metrics such as coverage and diversity. Compare user-level results as well as the mean: an aggregate score can hide poor experiences for particular users. section 4.6.3 section 4.10

How AI Recommender Systems covers it

How deep it goes
Defines the per-user score and mean, then implements first-match lookup and the no-match case at different cutoffs.
Examples
Uses first relevant items at positions 1 and 3, a no-result case, and a top-K metric-calculation listing.
What it assumes
Assumes ranked recommendations and relevance labels, while defining reciprocal rank from the first relevant position.
What you can do after
Evaluate/understand it by calculating top-K MRR and comparing a model with baselines and other quality measures.
How it connects
Contrasts MRR with precision, recall, NDCG, and MAP, and places it alongside coverage and diversity in evaluation.

Also explained in

Compare how 5 books explain mean reciprocal rank →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Recommender Systems. The section links point to the parts of the book it draws on.