liveBook · Manning

Retrieval ranking

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Ranking is the ordering of alternatives according to a score, preference, relevance, or other criterion.

A retrieval system can find relevant documents yet place them too low to be useful. Retrieval ranking orders results by relevance, ideally putting the most useful documents first. In retrieval-augmented generation (RAG), low-ranked results may be truncated or diluted by less relevant content within the language model’s context. section 6.2.3

Set-based precision and recall do not distinguish between rankings of the same retrieved documents. Rank-aware metrics help evaluate whether relevant results appear early enough to be useful. section 6.2.3

Two ways to evaluate a ranked list are:

  • Precision@k: divide the number of relevant documents in the top k results by k. This measures relevance within a cutoff, but does not distinguish their positions within that cutoff. section 6.2.3
  • Mean average precision (MAP): calculate average precision for each query by averaging precision at the retrieved relevant documents, then average those query scores. It rewards rankings that place relevant documents higher. section 6.2.3

Relevant documents ranked third and fourth

A financial research query retrieves five documents. The Q3 earnings report and risk factors section are relevant, but appear at ranks 3 and 4. Precision@3 is 1/3; Precision@5 is 2/5, or 0.40. Both documents were retrieved, but the most relevant information is not at the top. section 6.2.3

Early placement is not the only context consideration. Models can use information at the beginning or end of an input better than information buried in the middle, so a retrieval ranking should also be interpreted alongside how documents enter the context. section 6.2.3

Ranking also supports evaluation beyond retrieval. For example, a translation ranking model learns from judgments that one candidate is better than another, without requiring an absolute quality score. Comparing an automated metric’s ordering with human preferences can indicate whether the metric captures what people value, although human judgments are themselves imperfect. section 3.3.2 section 4.1.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains rank-aware evaluation through precision at a cutoff and mean average precision, with a worked retrieval example.
Examples
Uses a financial research query whose relevant earnings and risk documents appear third and fourth among five results.
What you can do after
Evaluate a retrieval ranking using precision@k and MAP, interpreting the results alongside context placement.
How it connects
Connects ranking to precision, recall, context windows, correlation with human judgments, and reward modeling.

Also explained in

Compare how 9 books explain retrieval ranking →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.