Summarization condenses a source document while preserving its essential information. Because many different phrasings can capture the same information, evaluating summaries rewards coverage rather than exact wording. section 2.2
ROUGE evaluates summaries through lexical overlap, emphasizing recall to measure coverage. Its variants and F-measure offer complementary views, but lexical matching cannot assess meaning, synonyms, or coherence. section 2.2.3
Different wording, shared information
Three summaries of a company’s earnings report describe record profits, earnings above expectations, or strong product demand driving results. Their phrasing differs, but each captures key information that exact-word matching could miss. section 2.2
|
Design Principle
|
ROUGE Implementation
|
Applied
|
|
Match metric to task
|
Recall for coverage-focused tasks.
|
Ask “What does success look like?” before choosing formulas.
|
|
Precision vs Recall
|
Recall-oriented for summarization.
|
Conservative outputs -> precision; Comprehensive outputs -> recall
|
|
Build metric families
|
ROUGE-N, -L, -W, -S variants
|
Multiple complementary metrics reveal more than any single score.
|
|
Use harmonic mean (F-measure)
|
Balances precision and recall
|
When both objectives must be satisfied, not trade off.
|
|
Allow appropriate flexibility
|
LCS permits non-contiguous matches
|
Match algorithm strictness to acceptable variations.
|
|
Reward sustained quality
|
ROUGE-W’s superlinear weighting
|
Consecutive success > sporadic success
|
|
Capture multiple distances
|
Skip-bigram for long-range
|
Local + distant relationships both matter
|
Compare ROUGE design choices for matching metrics to task goals and handling different kinds of overlap. section 2.2.3
















