A skip-bigram is an ordered pair of words (wᵢ, wⱼ) where i < j. Unlike an ordinary bigram, the words need not be adjacent, so the pair preserves order while allowing any number of intervening words. section 2.2.2
Build skip-bigrams by following these steps:
- Take a tokenized sequence such as “The cat sat on the mat.” section 2.2.2
- Pair each word with every later word, including adjacent pairs such as (The, cat) and separated pairs such as (The, mat). section 2.2.2
- Compare candidate and reference skip-bigrams to measure ordered word-pair overlap in ROUGE-S. section 2.2.2
Sentence restructuring
A reference stating “The Federal Reserve raised interest rates” and a candidate stating “Rates were raised by the Federal Reserve” can retain ordered relationships such as “Federal” and “rates” even after restructuring. This makes skip-bigrams useful for summaries whose valid wording changes without removing the underlying lexical relationships. section 2.2.2
Table 2.3 ROUGE metric variants and their use cases
|
Metric
|
What it means
|
Best used when
|
Limitations
|
|
ROUGE-N (N=1, 2)
|
N-gram recall. How many reference n-grams appear in the candidate?
|
Quick evaluations; comparing systems on the same datasets; need interpretable scores.
|
Misses paraphrasing; ROUGE-2 often zero for short texts.
|
|
ROUGE-L
|
Longest common subsequence (LCS). Word order preservation with gaps allowed
|
Evaluating restructured content, paraphrasing detection, and flexible word positioning matters.
|
Does not distinguish consecutive vs scattered matches.
|
|
ROUGE-W
|
Weighted LCS. Rewards consecutive matches more than scattered ones.
|
Evaluating fluency, phrase structure preservation, and distinguishing coherent from fragmented text.
|
More complex to implement; requires tuning the weighting function
|
|
ROUGE-S
|
Skip-bigram co-occurrence. Word pairs maintain order but not adjacency.
|
Long documents; capturing long-range dependencies; content where related concepts are separated.
|
Combinatorial explosion for long texts; may reward coincidental word pairs.
|
|
ROUGE-SU
|
Skip-bigram + unigram. ROUGE-S with unigram fallback.
|
Short texts where ROUGE-S might return zero; need robust scores across varying lengths,
|
Same limitations as ROUGE-S, plus potential noise from unigram.
|
Compare ROUGE-S with ROUGE-N, ROUGE-L, and ROUGE-W to choose between adjacency, flexible order, weighted phrase continuity, and long-range ordered matching. section 2.2.2
