liveBook · Manning

Skip-bigram

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A skip-bigram is an ordered pair of words from a sequence that may have any number of intervening words.

A skip-bigram is an ordered pair of words (wᵢ, wⱼ) where i < j. Unlike an ordinary bigram, the words need not be adjacent, so the pair preserves order while allowing any number of intervening words. section 2.2.2

Build skip-bigrams by following these steps:

  • Take a tokenized sequence such as “The cat sat on the mat.” section 2.2.2
  • Pair each word with every later word, including adjacent pairs such as (The, cat) and separated pairs such as (The, mat). section 2.2.2
  • Compare candidate and reference skip-bigrams to measure ordered word-pair overlap in ROUGE-S. section 2.2.2

Sentence restructuring

A reference stating “The Federal Reserve raised interest rates” and a candidate stating “Rates were raised by the Federal Reserve” can retain ordered relationships such as “Federal” and “rates” even after restructuring. This makes skip-bigrams useful for summaries whose valid wording changes without removing the underlying lexical relationships. section 2.2.2

Table 2.3 ROUGE metric variants and their use cases
Metric
What it means
Best used when
Limitations
ROUGE-N (N=1, 2)
N-gram recall. How many reference n-grams appear in the candidate?
Quick evaluations; comparing systems on the same datasets; need interpretable scores.
Misses paraphrasing; ROUGE-2 often zero for short texts.
ROUGE-L
Longest common subsequence (LCS). Word order preservation with gaps allowed
Evaluating restructured content, paraphrasing detection, and flexible word positioning matters.
Does not distinguish consecutive vs scattered matches.
ROUGE-W
Weighted LCS. Rewards consecutive matches more than scattered ones.
Evaluating fluency, phrase structure preservation, and distinguishing coherent from fragmented text.
More complex to implement; requires tuning the weighting function
ROUGE-S
Skip-bigram co-occurrence. Word pairs maintain order but not adjacency.
Long documents; capturing long-range dependencies; content where related concepts are separated.
Combinatorial explosion for long texts; may reward coincidental word pairs.
ROUGE-SU
Skip-bigram + unigram. ROUGE-S with unigram fallback.
Short texts where ROUGE-S might return zero; need robust scores across varying lengths,
Same limitations as ROUGE-S, plus potential noise from unigram.

Compare ROUGE-S with ROUGE-N, ROUGE-L, and ROUGE-W to choose between adjacency, flexible order, weighted phrase continuity, and long-range ordered matching. section 2.2.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines pair generation and illustrates matching, but does not provide a complete scoring equation or implementation.
Examples
Uses “The cat sat on the mat” and a Federal Reserve sentence to demonstrate adjacent, distant, and restructured pairs.
What it assumes
Assumes words can be ordered in a sequence, while defining skip-bigrams without requiring prior metric knowledge.
What you can do after
Apply via ROUGE-S to compare candidate and reference summaries while retaining ordered relationships across intervening words.
How it connects
Connects skip-bigrams to ROUGE-S, ROUGE-SU, n-grams, paraphrasing, word order, and summarization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.