liveBook · Manning

N-gram

As explained in LLM Evaluation and Alignment, The Foundational Ideas

An n-gram is a contiguous sequence of n tokens, words, or characters extracted from text.

An n-gram is a contiguous sequence of n items, typically words or tokens. A unigram contains one item, a bigram contains two consecutive items, and a trigram contains three. Larger n-grams capture longer sequences, but are harder to match across texts. section 2.1.2

N-gram overlap helps compare generated text with a reference. In translation evaluation, precision asks what fraction of candidate n-grams occur in a reference. In summarization, ROUGE-N instead measures recall: how many reference n-grams occur in the candidate. ROUGE-N limits repeated matches to their co-occurrence count, preventing repetition from artificially inflating the score. section 2.1.2 section 2.2.2

Unigram and bigram recall

Reference: “Google announced new AI features for search.” Candidate: “Google revealed AI search capabilities.” Three of the seven reference unigrams match: Google, AI, and search. ROUGE-1 recall is 3/7, or about 0.429. None of the six reference bigrams match, so ROUGE-2 recall is 0/6, or 0. The scores reveal shared words but no shared adjacent word pairs. section 2.2.2

Exact n-gram matching can miss paraphrases, and longer matches can be rare in short texts. A longest common subsequence offers a different comparison: it preserves relative order but allows gaps between matching words. section 2.2.2

N-grams also help investigate data contamination: overlap between evaluation data and training data that materially affects measured performance. Dataset-level checks can use hashed n-gram overlap or other exact-match heuristics. In the GPT-3 evaluation protocol, benchmark examples with 13-gram overlap with pretraining data were removed to create clean subsets. High BLEU or ROUGE alone does not prove contamination; constrained answers or canonical phrasing can also produce high overlap. section 2.3.4

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines n-gram orders and works through recall calculations, while describing contamination checks as heuristics rather than implementations.
Examples
Calculates unigram and bigram recall for candidate and reference summaries and cites 13-gram filtering in GPT-3 benchmark evaluation.
What you can do after
Evaluate overlap scores in context, distinguishing shared words from shared phrases and avoiding contamination claims based on high scores alone.
How it connects
Connects n-grams to precision, recall, BLEU, ROUGE-N, and longest common subsequence.

Also explained in

Compare how 6 books explain n-gram →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.