liveBook · Manning

Mean reciprocal rank

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Mean reciprocal rank is the mean of the reciprocal ranks of the first relevant result across a set of queries.

Mean reciprocal rank (MRR) measures how quickly a retrieval system surfaces the first relevant result. For each query, it uses the reciprocal of that result’s rank, then averages across queries. section 6.2.3

A relevant result at rank 3

If the first relevant document appears at rank 3, its reciprocal rank is 1/3, approximately 0.33. This makes MRR useful when one good document is often sufficient to answer a question. section 6.2.3

Choose MRR according to how retrieved context is used:

  • Use MRR when any one relevant document can answer the query. section 6.2.3
  • Prefer Precision@k when a fixed number of documents enters the context window. section 6.2.3
  • Use nDCG when all relevant documents matter and relevance has graded levels. section 6.2.3
  • Interpret MRR alongside precision and recall rather than treating it as a complete retrieval diagnosis. section 6.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines first-result reciprocal ranking and contrasts metric selection, but does not develop a full implementation procedure.
Examples
Uses a query whose first relevant document appears at rank 3, producing an MRR contribution of approximately 0.33.
What it assumes
Assumes understanding of ranked retrieval results and relevant documents, while explaining the metric’s practical selection from that basis.
What you can do after
Choose MRR when one relevant document suffices, and select alternative metrics for fixed-context or graded-relevance retrieval.
How it connects
Connects MRR with Precision@k, MAP, nDCG, precision, recall, and retrieval-augmented generation.

Also explained in

Compare how 5 books explain mean reciprocal rank →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.