liveBook · Manning

BLEU score

As explained in LLM Evaluation and Alignment, The Foundational Ideas

BLEU is an automated machine-translation metric that compares generated text with human reference translations using matching n-grams.

BLEU (Bilingual Evaluation Understudy) is an automatic metric for machine translation. It compares a generated translation, called the candidate, with one or more human reference translations. Matching words and phrases provide a fast, repeatable proxy for translation quality, not a direct measure of meaning. section 2.0 section 2.1.1

To calculate a BLEU score:

  • Count matching candidate n-grams—sequences of words—at several lengths. Modified precision clips match counts to prevent repeated words from inflating the score. section 2.1.1 section 2.1.8
  • Combine the n-gram precisions using a geometric mean, which requires balanced performance across the different lengths. section 2.1.7 section 2.1.8
  • Apply a brevity penalty when the candidate is too short. The result ranges from 0 to 1 and is often reported on a 0–100 scale. section 2.1.7 section 2.1.8

A translation comparison

For an 18-word candidate compared with a 16-word reference, unigram precision is 13/18 and four-gram precision is 3/15. Combining precisions for one- through four-word sequences and applying no brevity penalty gives a BLEU score of 0.382. Some correct meaning is expressed with different wording, reducing overlap. section 2.1.7

Table 2.1 Example calculations for n-gram precision
N-gram
Candidate n-grams
Word matches
Precision
Unigram (n=1)
18
13
13/18 = 0.722
Bigram (n=2)
17
8
8/17 = 0.471
Trigram (n=3)
16
5
5/16 = 0.313
4-gram (n=4)
15
3
3/15 = 0.200

The complete precision counts show how matches decrease as the word sequences get longer. section 2.1.7

Interpret scores within the same evaluation setup: language pair, domain, and reference quality affect results. A low score can expose mismatched phrases, but valid paraphrases may be penalized. Conversely, lexical overlap alone cannot establish that a translation preserves meaning. section 2.1.7 section 1.1.2 section 2.0

BLEU is useful for translation benchmarks and inexpensive regression checks. A score drop after a system change flags a change relative to references; it does not explain whether meaning improved or worsened. Follow up with human review or targeted evaluation. For open-ended outputs, consider semantic metrics such as BERTScore or COMET alongside lexical checks. section 2.3.1 section 2.3.3 section 3.4.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Works through n-gram precisions, count clipping, geometric aggregation, and a brevity penalty to produce a score.
Examples
Calculates a 0.382 score for an English candidate translation against a human reference.
What you can do after
Evaluate translation scores in context and diagnose when lexical overlap needs human or semantic follow-up.
How it connects
Connects lexical precision and reference-based evaluation with ROUGE, BERTScore, COMET, and regression testing.

Also explained in

Compare how 5 books explain BLEU score →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.