liveBook · Manning

Precision

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Precision is the proportion of predicted or candidate items that satisfy the matching criterion, here referring to overlapping words or segments.

Precision asks: how much of a system’s output meets the matching criterion? For text generation, it is the number of candidate words or n-grams found in a reference, divided by the total number of candidate words or n-grams. An n-gram is a contiguous sequence of n tokens. section 2.1.2

Counting word matches

For the candidate “cat is missing” and the reference “The cat is on the mat,” two of the candidate’s three words match. The unigram precision is 2/3. The score says how much of the candidate overlaps, not whether the candidate expresses the reference’s full meaning. section 2.1.2 section 2.0

Naive word counting can be misleading. A candidate that repeats “the” seven times overlaps with references containing “the,” but is not a useful translation. BLEU addresses evaluation through modified n-gram precision at several n-gram lengths, combining the scores and applying a brevity penalty. section 2.1.2 section 2.1.1

Figure 2.1 The BLEU evaluation pipeline from input to final score. BLEU takes generated candidates and references, calculates word overlaps for different word segment lengths, combines these scores using a geometric mean, and applies a length penalty to produce the final score.

Follow how BLEU combines n-gram overlap scores and a length penalty into a final score. section 2.1.1

Recall reverses the question: how much of the reference appears in the candidate? A short output can have high precision while missing important information. ROUGE emphasizes recall for summarization, and an F-measure combines precision and recall when both matter. section 2.2.1 section 2.2.2

Choose the metric for the task. Precision favors outputs that avoid unmatched content; recall favors coverage. Exact lexical overlap can miss valid paraphrases, so a high or low score alone is not a universal judgment of quality. Precision also appears in retrieval evaluation, where the question is whether retrieved documents are relevant. section 2.2 section 6.0 section 2.0

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines n-gram precision, illustrates a repetition failure, and places modified precision within BLEU’s scoring pipeline.
Examples
Uses a translation repeating “the” and contrasts short summaries with comprehensive ones.
What you can do after
Evaluate and understand overlap scores in context, including repetition, missed coverage, and paraphrasing limitations.
How it connects
Connects precision to recall, BLEU, ROUGE, n-grams, and the F-measure.

Also explained in

Compare how 27 books explain precision →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.