BLEU score
BLEU is an automatic text-generation evaluation metric that compares candidate text with reference text using modified n-gram precision and a brevity penalty.
LLM Evaluation and Alignment, The Foundational Ideas
BLEU is an automated machine-translation metric that compares generated text with human reference translations using matching n-grams.
- How it is used
- Automates reference-based translation evaluation and supplies inexpensive regression signals for system changes.
- How deep it goes
- Works through n-gram precisions, count clipping, geometric aggregation, and a brevity penalty to produce a score.
- Examples
- Calculates a 0.382 score for an English candidate translation against a human reference.
- What you can do after
- Evaluate translation scores in context and diagnose when lexical overlap needs human or semantic follow-up.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains BLEU score →
Generative AI in Action
BLEU is an automatic metric that evaluates machine-translated text by measuring its n-gram overlap with a human reference translation.
- How it is used
- Uses n-gram overlap to quantify translation quality during model fine-tuning and broader text-generation evaluation.
- How deep it goes
- Explains n-gram precision, score range, brevity penalty, and length ratio, but stops short of mathematical derivation.
- Examples
- Demonstrates BLEU 0.047 with n-gram precisions, a brevity penalty, length ratio, and comparison against ROUGE and BERTScore.
- What you can do after
- Evaluate generated text with the Evaluate library and interpret overlap, length, and phrase-formation signals alongside other metrics.
Sutskever's List
BLEU is an automatic machine-translation metric based on n-gram overlap between generated and reference translations.
- How it is used
- Evaluates how closely machine-generated translations resemble human-written translations.
- How deep it goes
- Defines n-gram-overlap evaluation and interprets scores, but does not explain the metric’s internal calculation.
- Examples
- Uses English-to-French WMT results, comparing BLEU scores of 25.9 and 29.03 after dropout regularization.
- What you can do after
- Evaluate translation quality by interpreting higher BLEU scores as closer resemblance to human-written references.
LLMs in Production
BLEU is an evaluation metric for machine translation that measures modified n-gram precision against one or more reference translations.
- How it is used
- Evaluates machine-translation predictions against reference translations.
- How deep it goes
- Explains precision failure, brevity penalty, modified n-gram precision, and a sentence-level calculation without deriving the formula.
- Examples
- Uses the cat-and-hat precision example and a Zelda translation comparison with two references.
- What you can do after
- Apply it via NLTK to calculate sentence BLEU for a prediction and multiple tokenized references.
Deep Learning with R, Third Edition
A BLEU score measures the similarity of machine-translated text to one or more high-quality reference translations using n-gram overlap.
- How it is used
- Evaluates machine-translation quality when generated sequences cannot be judged reliably by next-token accuracy.
- How deep it goes
- Defines BLEU and motivates its use but stops short of explaining its n-gram calculation or interpreting score values.
- Examples
- Uses English-to-Spanish test translations, including reordered or erroneous outputs, to illustrate real inference behavior.
- What you can do after
- Evaluate machine-translation outputs with a metric more appropriate than teacher-forced next-token accuracy.
Read how Deep Learning with R, Third Edition explains BLEU score →
Also covered in
- Transformers in Action
- Domain-Specific Small Language Models
- Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI
What to know first
- machine translationMachine translation is the automated computational conversion of text or speech from one natural language into another.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- summarizationSummarization is the production of a shorter representation of source information that preserves its important content or meaning.
Related concepts
- geometric meanuses · A geometric mean is the positive nth root of the product of n positive values, representing their multiplicative average.
- machine translationmeasures · Machine translation is the automated computational conversion of text or speech from one natural language into another.
- n-gramuses · An n-gram is a contiguous sequence of n consecutive items, such as words, tokens, or characters, drawn from a larger sequence.
- ROUGEcontrasts with · ROUGE is a family of automatic text-generation evaluation metrics that compare generated text with reference text using lexical overlap, often emphasizing recall.
Questions about BLEU score
What is BLEU score?
BLEU is an automatic text-generation evaluation metric that compares candidate text with reference text using modified n-gram precision and a brevity penalty.
Which Manning books explain BLEU score?
LLM Evaluation and Alignment, The Foundational Ideas, Generative AI in Action, Sutskever's List, LLMs in Production and Deep Learning with R, Third Edition.
What should I know before learning BLEU score?
Manning books that teach BLEU score usually introduce machine translation, accuracy, language model and summarization first.
What is the difference between BLEU score and ROUGE?
BLEU is an automatic text-generation evaluation metric that compares candidate text with reference text using modified n-gram precision and a brevity penalty. ROUGE is a family of automatic text-generation evaluation metrics that compare generated text with reference text using lexical overlap, often emphasizing recall.
What is BLEU score also called?
BLEU score is also called Bilingual Evaluation Understudy.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




