liveBook · Manning

Precision

As explained in Retrieval Augmented Generation, The Foundational Ideas

Precision is the proportion of retrieved items that are relevant according to the evaluation criterion.

Precision measures the proportion of retrieved items that are relevant: overlap between retrieved and relevant items divided by everything returned. In retrieval systems, Precision@k evaluates the first k results. section 14.1.1

Compliance retrieval

A financial-services assistant retrieves five documents for a compliance query; three are relevant. Its Precision@5 is 3/5, or 0.6. This does not show whether the relevant documents appear early enough for the generator to use them. section 14.1.1

Precision is useful but incomplete for RAG evaluation. It ignores rank position and whether relevance supplies the specific information needed for generation. Reranking and hybrid search can improve retrieval quality, so measure precision alongside answer quality. section 14.1.1 section 4.6

Choose the retrieval emphasis according to the downstream reader:

  • Optimize for precision when the reader handles only a few documents and the top passages must contain the answer. section 5.3.1
  • Shift toward recall when a FiD reader can process many passages and synthesize information across the retrieved set, while monitoring added noise. section 5.3.1
Figure 14.1 Precision and recall. Of the items returned by the retriever (the selected set), precision measures the fraction that are relevant. Of all relevant items in the corpus, recall measures the fraction that the retriever surfaced.

Compare precision's selected-set denominator with recall's corpus-wide denominator using the retrieved and relevant item sets. section 14.1.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines Precision@k and calculates it numerically, but defers rank position and generation utility to other metrics.
Examples
Uses a financial compliance query retrieving five documents, three relevant, for Precision@5 = 0.6.
What it assumes
Assumes retrieved and relevant item sets, while defining their overlap-based ratio from the foundation.
What you can do after
Evaluate RAG retrieval cleanliness with Precision@k and compare it with answer quality and rank-aware measures.
How it connects
Connects precision with recall, F1 score, MRR, NDCG, MAP, reranking, hybrid search, and RAG.

Also explained in

Compare how 27 books explain precision →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.