BLEU (Bilingual Evaluation Understudy) is an automatic metric for machine translation. It compares a generated translation, called the candidate, with one or more human reference translations. Matching words and phrases provide a fast, repeatable proxy for translation quality, not a direct measure of meaning. section 2.0 section 2.1.1
To calculate a BLEU score:
- Count matching candidate n-grams—sequences of words—at several lengths. Modified precision clips match counts to prevent repeated words from inflating the score. section 2.1.1 section 2.1.8
- Combine the n-gram precisions using a geometric mean, which requires balanced performance across the different lengths. section 2.1.7 section 2.1.8
- Apply a brevity penalty when the candidate is too short. The result ranges from 0 to 1 and is often reported on a 0–100 scale. section 2.1.7 section 2.1.8
A translation comparison
For an 18-word candidate compared with a 16-word reference, unigram precision is 13/18 and four-gram precision is 3/15. Combining precisions for one- through four-word sequences and applying no brevity penalty gives a BLEU score of 0.382. Some correct meaning is expressed with different wording, reducing overlap. section 2.1.7
Table 2.1 Example calculations for n-gram precision
|
N-gram
|
Candidate n-grams
|
Word matches
|
Precision
|
|
Unigram (n=1)
|
18
|
13
|
13/18 = 0.722
|
|
Bigram (n=2)
|
17
|
8
|
8/17 = 0.471
|
|
Trigram (n=3)
|
16
|
5
|
5/16 = 0.313
|
|
4-gram (n=4)
|
15
|
3
|
3/15 = 0.200
|
The complete precision counts show how matches decrease as the word sequences get longer. section 2.1.7
Interpret scores within the same evaluation setup: language pair, domain, and reference quality affect results. A low score can expose mismatched phrases, but valid paraphrases may be penalized. Conversely, lexical overlap alone cannot establish that a translation preserves meaning. section 2.1.7 section 1.1.2 section 2.0
BLEU is useful for translation benchmarks and inexpensive regression checks. A score drop after a system change flags a change relative to references; it does not explain whether meaning improved or worsened. Follow up with human review or targeted evaluation. For open-ended outputs, consider semantic metrics such as BERTScore or COMET alongside lexical checks. section 2.3.1 section 2.3.3 section 3.4.1




