Precision measures how many predicted items are correct among all predicted items. For text evaluation, it can reward short predictions too generously: predicting “cat hat” against “the cat in the hat” gives perfect precision despite missing most of the target. section 4.2.1
BLEU adapts precision for machine translation by adding a brevity penalty and modified N-gram precision. These adjustments penalize predictions that are too short and support comparison against multiple targets. section 4.2.1
A misleading perfect score
With target “the cat in the hat” and prediction “cat hat,” standard precision is 1.0 because both predicted words occur in the target, while recall is only 0.4. BLEU addresses this imbalance. section 4.2.1
Listing 4.2 Using BLEU
import nltk.translate.bleu_score as bleu
target = [
"The game 'The Legend of Zelda' follows the adventures of the \
hero Link in the magical world of Hyrule.".split(),
"Link goes on awesome quests and battles evil forces to \
save Princess Zelda and restore peace to Hyrule.".split(),
]
prediction = "Link embarks on epic quests and battles evil forces to \
save Princess Zelda and restore peace in the land of Hyrule.".split()
score = bleu.sentence_bleu(target, prediction)
print(score)
# 0.6187934993051339
Run a minimal NLTK BLEU example comparing one prediction with two Zelda-related target sentences. section 4.2.1
