liveBook · Manning

Mean reciprocal rank

As explained in Retrieval Augmented Generation, The Foundational Ideas

Mean reciprocal rank is the mean of the reciprocal ranks assigned to the first correct result across evaluation queries.

Mean reciprocal rank (MRR) evaluates how quickly a ranked retrieval system surfaces the first correct or relevant result. For each query, score 1 divided by the first relevant result's rank, then average those scores across queries. section 3.2.1 section 14.1.1

MRR rewards early retrieval sharply: rank 1 contributes 1.0, rank 2 contributes 0.5, and rank 50 contributes 0.02. This makes it useful when downstream generation sees only a few ranked documents or attends unevenly across the context. section 3.2.1 section 14.1.1

Two-query calculation

If the first relevant compliance document appears at rank 3 for one query and rank 1 for another, MRR = (1/3 + 1/1) / 2 = 0.667. An MRR of 1.0 means the top result is always relevant. section 14.1.1

Use MRR alongside other retrieval metrics because it emphasizes the first hit rather than the whole ranked set.

  • Recall@k records whether any correct document appears within the first k positions, even when it is buried; MRR penalizes that delay. section 3.2.1
  • NDCG@k considers multiple relevant documents and discounts their positions, whereas MRR considers only the first relevant result. section 14.1.1
  • Interpret gains with a baseline and check accuracy, latency, index size, dataset splits, and statistical evidence rather than treating MRR alone as sufficient. section 3.2.1
MRR=(1/|Q|)*∑i=1n1/ranki

The equation expresses MRR as the average reciprocal rank of the first relevant result across the query set. section 14.1.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines reciprocal-rank averaging, gives numerical examples, and contrasts MRR with Recall@k and NDCG@k without implementing it.
Examples
Uses rank-one, rank-two, and rank-50 contributions plus a two-query compliance-document calculation.
What it assumes
Assumes ranked retrieval and relevance judgments while explaining the reciprocal-rank calculation from basic examples.
What you can do after
Evaluate retrieval rankings with MRR and interpret its tradeoffs against Recall@k, NDCG@k, latency, and index size.
How it connects
Connects MRR to information retrieval, Recall@k, NDCG@k, precision, ranking, and retrieval-augmented generation.

Also explained in

Compare how 5 books explain mean reciprocal rank →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.