A retrieval system can find relevant documents yet place them too low to be useful. Retrieval ranking orders results by relevance, ideally putting the most useful documents first. In retrieval-augmented generation (RAG), low-ranked results may be truncated or diluted by less relevant content within the language model’s context. section 6.2.3
Set-based precision and recall do not distinguish between rankings of the same retrieved documents. Rank-aware metrics help evaluate whether relevant results appear early enough to be useful. section 6.2.3
Two ways to evaluate a ranked list are:
- Precision@k: divide the number of relevant documents in the top k results by k. This measures relevance within a cutoff, but does not distinguish their positions within that cutoff. section 6.2.3
- Mean average precision (MAP): calculate average precision for each query by averaging precision at the retrieved relevant documents, then average those query scores. It rewards rankings that place relevant documents higher. section 6.2.3
Relevant documents ranked third and fourth
A financial research query retrieves five documents. The Q3 earnings report and risk factors section are relevant, but appear at ranks 3 and 4. Precision@3 is 1/3; Precision@5 is 2/5, or 0.40. Both documents were retrieved, but the most relevant information is not at the top. section 6.2.3
Early placement is not the only context consideration. Models can use information at the beginning or end of an input better than information buried in the middle, so a retrieval ranking should also be interpreted alongside how documents enter the context. section 6.2.3
Ranking also supports evaluation beyond retrieval. For example, a translation ranking model learns from judgments that one candidate is better than another, without requiring an absolute quality score. Comparing an automated metric’s ordering with human preferences can indicate whether the metric captures what people value, although human judgments are themselves imperfect. section 3.3.2 section 4.1.1








