Reciprocal rank measures how quickly the first relevant item appears in a ranked list: 1 divided by its position, or 0 when no relevant item appears. section 4.6.3
Position determines the score
A relevant item at position 1 gives a score of 1.0; at position 3, 0.33; with no relevant item, the score is 0. section 4.6.3
Apply reciprocal rank within mean reciprocal rank (MRR):
- Compute each user's reciprocal rank, then average those scores across users. section 4.6.3
- Use it when users typically want one good result, such as search, question answering, or play-next recommendations. section 4.6.3
Listing 4.8 Calculate metrics for different K values
for k in k_values: #A
if k <= len(top_k_indices): #B
top_k_rel = relevance_scores[:k] #C
if np.sum(top_k_rel) > 0: #D
ideal_rel = np.sort(top_k_rel)[::-1] #E
ndcg_k = ndcg_score([ideal_rel], [top_k_rel])
all_metrics[k]['ndcg'].append(ndcg_k)
precision_k = np.sum(top_k_rel) / k #F
all_metrics[k]['precision'].append(precision_k)
if len(test_indices) > 0:
recall_k = np.sum(top_k_rel) / len(test_indices) #G
all_metrics[k]['recall'].append(recall_k)
first_relevant = np.where(top_k_rel == 1)[0] #H
if len(first_relevant) > 0:
mrr = 1.0 / (first_relevant[0] + 1)
all_metrics[k]['mrr'].append(mrr)
else:
all_metrics[k]['mrr'].append(0.0)
if np.sum(top_k_rel) > 0: #I
ap = 0.0
relevant_found = 0
for i, rel in enumerate(top_k_rel):
if rel == 1:
relevant_found += 1
ap += relevant_found / (i + 1)
ap /= np.sum(top_k_rel)
all_metrics[k]['map'].append(ap)
else:
all_metrics[k]['map'].append(0.0)The metric calculation checks for the first relevant item and assigns the reciprocal of its one-based position, otherwise assigning zero. section 4.6.3
