BLEU score
BLEU is an automatic text-generation evaluation metric that compares candidate text with reference text using modified n-gram precision and a brevity penalty.
Generative AI in Action
BLEU is an automatic metric that evaluates machine-translated text by measuring its n-gram overlap with a human reference translation.
- How it is used
- Uses n-gram overlap to quantify translation quality during model fine-tuning and broader text-generation evaluation.
- How deep it goes
- Explains n-gram precision, score range, brevity penalty, and length ratio, but stops short of mathematical derivation.
- Examples
- Demonstrates BLEU 0.047 with n-gram precisions, a brevity penalty, length ratio, and comparison against ROUGE and BERTScore.
- What you can do after
- Evaluate generated text with the Evaluate library and interpret overlap, length, and phrase-formation signals alongside other metrics.
Sutskever's List
BLEU is an automatic machine-translation metric based on n-gram overlap between generated and reference translations.
- How it is used
- Evaluates how closely machine-generated translations resemble human-written translations.
- How deep it goes
- Defines n-gram-overlap evaluation and interprets scores, but does not explain the metric’s internal calculation.
- Examples
- Uses English-to-French WMT results, comparing BLEU scores of 25.9 and 29.03 after dropout regularization.
- What you can do after
- Evaluate translation quality by interpreting higher BLEU scores as closer resemblance to human-written references.
LLMs in Production
BLEU is an evaluation metric for machine translation that measures modified n-gram precision against one or more reference translations.
- How it is used
- Evaluates machine-translation predictions against reference translations.
- How deep it goes
- Explains precision failure, brevity penalty, modified n-gram precision, and a sentence-level calculation without deriving the formula.
- Examples
- Uses the cat-and-hat precision example and a Zelda translation comparison with two references.
- What you can do after
- Apply it via NLTK to calculate sentence BLEU for a prediction and multiple tokenized references.
Deep Learning with R, Third Edition
A BLEU score measures the similarity of machine-translated text to one or more high-quality reference translations using n-gram overlap.
- How it is used
- Evaluates machine-translation quality when generated sequences cannot be judged reliably by next-token accuracy.
- How deep it goes
- Defines BLEU and motivates its use but stops short of explaining its n-gram calculation or interpreting score values.
- Examples
- Uses English-to-Spanish test translations, including reordered or erroneous outputs, to illustrate real inference behavior.
- What you can do after
- Evaluate machine-translation outputs with a metric more appropriate than teacher-forced next-token accuracy.
Read how Deep Learning with R, Third Edition explains BLEU score →
Related concepts
- machine translationmeasures · Machine translation is the automated computational conversion of text or speech from one natural language into another.
- ROUGEcontrasts with · ROUGE is a family of automatic text-generation evaluation metrics that compare generated text with reference text using lexical overlap, often emphasizing recall.
- n-gramuses · An n-gram is a contiguous sequence of n consecutive items, commonly linguistic tokens or characters, drawn from a larger sequence.
- mathematical equivalencecontrasts with · Mathematical equivalence is a relation in which different expressions or representations have the same value or meaning under a specified interpretation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.