In a “play next” recommendation, finding one good option quickly may matter more than finding every relevant option. Mean reciprocal rank (MRR) measures how early the first relevant item appears. For each user, divide 1 by that item's one-based position, then average those scores across users. If no relevant item appears, that user's score is 0. section 4.6.3
Three recommendation lists
Suppose the first relevant item appears at position 1 for one user and position 3 for another, while a third user sees none. Their reciprocal ranks are 1, about 0.33, and 0; MRR is about 0.44. Moving the second user's first relevant item higher raises the score. section 4.6.3
To calculate MRR at a chosen cutoff K, take each user's top-K relevance labels, find the first label equal to 1, and convert its zero-based index to a one-based rank before taking the reciprocal. Record 0 when there is no match. Changing K can change which users have a match; ranking metrics often plateau once relevant items have been surfaced. section 4.6.3 section 4.6
Listing 4.8 Calculate metrics for different K values
for k in k_values: #A
if k <= len(top_k_indices): #B
top_k_rel = relevance_scores[:k] #C
if np.sum(top_k_rel) > 0: #D
ideal_rel = np.sort(top_k_rel)[::-1] #E
ndcg_k = ndcg_score([ideal_rel], [top_k_rel])
all_metrics[k]['ndcg'].append(ndcg_k)
precision_k = np.sum(top_k_rel) / k #F
all_metrics[k]['precision'].append(precision_k)
if len(test_indices) > 0:
recall_k = np.sum(top_k_rel) / len(test_indices) #G
all_metrics[k]['recall'].append(recall_k)
first_relevant = np.where(top_k_rel == 1)[0] #H
if len(first_relevant) > 0:
mrr = 1.0 / (first_relevant[0] + 1)
all_metrics[k]['mrr'].append(mrr)
else:
all_metrics[k]['mrr'].append(0.0)
if np.sum(top_k_rel) > 0: #I
ap = 0.0
relevant_found = 0
for i, rel in enumerate(top_k_rel):
if rel == 1:
relevant_found += 1
ap += relevant_found / (i + 1)
ap /= np.sum(top_k_rel)
all_metrics[k]['map'].append(ap)
else:
all_metrics[k]['map'].append(0.0)The metric loop shows the first-relevant-item lookup, index adjustment, and zero-score branch alongside other top-K metrics. section 4.6.3
MRR answers a narrower question than precision or recall, which count relevant items, and MAP, which rewards placing multiple relevant items near the top. Evaluate it against relevance sets built from historical data, alongside baselines and metrics such as coverage and diversity. Compare user-level results as well as the mean: an aggregate score can hide poor experiences for particular users. section 4.6.3 section 4.10




