liveBook · Manning

Precision@k

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Precision@k is the proportion of relevant results among the first k results in a ranked retrieval list.

Precision@k measures the proportion of relevant documents among the first k results in a ranked retrieval list. In RAG, k usually equals the number of documents inserted into the context window, so the metric evaluates what can influence generation. section 6.2.3 section 6.5

precision@k = number of relevant documents in top k / k

Compute Precision@k by dividing the number of relevant documents in the top k positions by k. section 6.2.3

Financial research retrieval

Suppose five documents are retrieved, with the Q3 earnings report ranked third and the risk-factors section ranked fourth. Precision@3 is 1/3 because only one relevant document appears in the first three results; Precision@5 is 2/5, or 0.40. section 6.2.3

Precision@k exposes whether relevant material is buried below the context cutoff, but it does not distinguish different orderings within the top k. Use MAP, MRR, or nDCG when finer rank sensitivity is needed. section 6.2.3 section 6.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines the formula and works through top-three and top-five calculations, but does not provide implementation code.
Examples
Uses a five-document financial research ranking with relevant earnings and risk-factors documents at ranks three and four.
What it assumes
Assumes ranked retrieval results and relevance judgments while defining the top-k calculation directly.
What you can do after
Evaluate retrieval quality by calculating the relevant-document share within a chosen context cutoff.
How it connects
Connects precision with recall, context windows, MAP, MRR, nDCG, and RAG retrieval evaluation.

Also explained in

Compare how 3 books explain Precision@k →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.