liveBook · Manning

BLEU score

As explained in LLMs in Production

BLEU is an evaluation metric for machine translation that measures modified n-gram precision against one or more reference translations.

BLEU evaluates machine-translation predictions by comparing their n-grams with one or more reference translations. Unlike ROUGE, it emphasizes precision, while correcting precision’s tendency to reward short, incomplete outputs. section 4.2.1

BLEU makes two practical adjustments:

  • Applies a brevity penalty when the prediction is shorter than the target. section 4.2.1
  • Uses modified n-gram precision to compare a prediction against multiple reference translations. section 4.2.1

Why ordinary precision can mislead

For target “the cat in the hat” and prediction “cat hat,” ordinary precision reaches 1.0 because both predicted words occur in the target, despite the incomplete output. BLEU penalizes this shortness. In a Zelda comparison with two references, NLTK produces a sentence BLEU score of about 0.619. section 4.2.1

How LLMs in Production covers it

How deep it goes
Explains precision failure, brevity penalty, modified n-gram precision, and a sentence-level calculation without deriving the formula.
Examples
Uses the cat-and-hat precision example and a Zelda translation comparison with two references.
What you can do after
Apply it via NLTK to calculate sentence BLEU for a prediction and multiple tokenized references.
How it connects
Contrasts BLEU’s precision orientation with ROUGE’s recall orientation and connects both through n-gram comparison.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

Compare how 4 books explain BLEU score →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.