liveBook · Manning

Summarization

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Summarization is the task of condensing a source document into a shorter text that preserves its essential information.

Summarization condenses a source document while preserving its essential information. Because many different phrasings can capture the same information, evaluating summaries rewards coverage rather than exact wording. section 2.2

ROUGE evaluates summaries through lexical overlap, emphasizing recall to measure coverage. Its variants and F-measure offer complementary views, but lexical matching cannot assess meaning, synonyms, or coherence. section 2.2.3

Different wording, shared information

Three summaries of a company’s earnings report describe record profits, earnings above expectations, or strong product demand driving results. Their phrasing differs, but each captures key information that exact-word matching could miss. section 2.2

Design Principle
ROUGE Implementation
Applied
Match metric to task
Recall for coverage-focused tasks.
Ask “What does success look like?” before choosing formulas.
Precision vs Recall
Recall-oriented for summarization.
Conservative outputs -> precision; Comprehensive outputs -> recall
Build metric families
ROUGE-N, -L, -W, -S variants
Multiple complementary metrics reveal more than any single score.
Use harmonic mean (F-measure)
Balances precision and recall
When both objectives must be satisfied, not trade off.
Allow appropriate flexibility
LCS permits non-contiguous matches
Match algorithm strictness to acceptable variations.
Reward sustained quality
ROUGE-W’s superlinear weighting
Consecutive success > sporadic success
Capture multiple distances
Skip-bigram for long-range
Local + distant relationships both matter

Compare ROUGE design choices for matching metrics to task goals and handling different kinds of overlap. section 2.2.3

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains recall-oriented ROUGE and F-measure, while noting that lexical metrics miss meaning, synonyms, and coherence.
Examples
Uses three differently worded summaries of a company earnings report to illustrate shared information coverage.
What you can do after
Evaluate summaries with ROUGE by prioritizing information coverage and interpreting lexical-overlap scores cautiously.
How it connects
Connects summarization evaluation to ROUGE, BLEU, lexical overlap, recall, precision, and F-measure.

Also explained in

Compare how 17 books explain summarization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.