BLEU (Bilingual Evaluation Understudy) evaluates machine-translated text by comparing the candidate with one or more human reference translations. It measures content overlap rather than intelligibility or grammatical correctness, making it useful for quantitative translation evaluation and model fine-tuning. section 9.3.2 section 12.2.1
BLEU combines several signals:
- N-gram precision counts matching word sequences between the candidate translation and the reference, covering 1-gram through 4-gram matches in the worked evaluation. section 12.2
Interpreting an automated evaluation
A candidate text scored BLEU 0.047 (4.7%), with reasonable 1-gram precision but very few longer n-gram matches. Its brevity penalty was 1.0, while its length ratio of 1.27 showed that the candidate was 27% longer than the reference. The low BLEU score indicates weak phrase and sentence overlap, but comparison with BERTScore showed that surface matching and semantic similarity can lead to different conclusions.
Table 12.1 Traditional evaluation metrics
|
Metric
|
Focus
|
Method
|
Limitations
|
|---|---|---|---|
|
BLEU
|
It primarily measures precision, the percentage of words in the machine- generated text that appear in the reference text.
|
It compares n-grams (word sequences) of the candidate translation with the reference translation and counts the matches.
|
It can miss the mark on semantic meaning because it doesn’t account for synonyms or the context of words. It also doesn’t handle word reordering well.
|
|
ROUGE
|
It is more recall oriented, focusing on the percentage of words from the reference text that appear in the generated text.
|
It has several variants, such as ROUGE-N, which compares n-grams, and ROUGE-L, which looks at the longest common subsequence.
|
Like BLEU, ROUGE can overlook semantic similarities and paraphrasing because it’s based on exact word matches.
|
|
BERTScore
|
It evaluates semantic similarity rather than relying on exact word matches.
|
It uses contextual embeddings from models such as BERT to represent the text and calculates the cosine similarity between these embeddings.
|
It can capture paraphrasing and semantic meaning better than BLEU and ROUGE because it considers each word’s context.
|
Compare BLEU with ROUGE and BERTScore by focus, method, and limitations. section 12.2
BLEU can miss synonyms, semantic equivalence, context, and word reordering. Use it alongside qualitative measures such as fluency, coherence, conciseness, and factual accuracy rather than as a complete quality judgment. section 9.3.2
