BERTScore
BERTScore is an automatic text-generation evaluation metric that compares candidate and reference texts using contextual representations of their tokens.
LLM Evaluation and Alignment, The Foundational Ideas
A text-generation metric that compares semantic representations of candidate and reference text using contextual language-model embeddings.
- How it is used
- Evaluates generated text against references when valid paraphrases make word overlap too restrictive.
- How deep it goes
- Explains contextual token embeddings, pairwise cosine similarity, greedy precision and recall, optional IDF weighting, and baseline rescaling.
- Examples
- Compares “consumers prefer imported cars” with “people like foreign cars” and works through greedy matching for a weather paraphrase.
- What you can do after
- Evaluate and understand semantic similarity scores while recognizing factual errors, reference dependence, and encoder bias.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains BERTScore →
Building Specialized AI Systems
BERTScore is a text-generation evaluation metric that compares candidate and reference texts using contextual token representations from BERT-like models.
- How it is used
- Evaluates generated text when semantic equivalence matters more than exact word overlap.
- How deep it goes
- Explains contextual embeddings and token-level scoring, but defers model-specific implementation details and metric limitations beyond supplementation.
- Examples
- Uses generated and reference texts, with legal-document and customer-service evaluation needs as domain examples.
- What you can do after
- Evaluate text-generation quality with semantic similarity scores while supplementing them with expert and application-specific checks.
Read how Building Specialized AI Systems explains BERTScore →
Generative AI in Action
BERTScore is an automatic metric that evaluates generated text by comparing contextual semantic representations of candidate and reference text.
- How it is used
- Evaluates generated summaries or translations by comparing candidate text with human reference text through semantic similarity.
- How deep it goes
- Defines contextual embeddings, cosine similarity, token matching, precision, recall, and F1 without mathematical derivation.
- Examples
- Calculates scores for an automatically generated article summary using the Evaluate library and reports precision, recall, and F1.
- What you can do after
- Evaluate generated text for semantic similarity and interpret precision, recall, and F1 alongside other evaluation methods.
What to know first
- BLEU scoreBLEU is an automatic text-generation evaluation metric that compares candidate text with reference text using modified n-gram precision and a brevity penalty.
- ROUGEROUGE is a family of automatic text-generation evaluation metrics that compare generated text with reference text using lexical overlap, often emphasizing recall.
- BERTBERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
- evaluationEvaluation is the systematic assessment of a system, model, process, or outcome against relevant criteria or objectives using evidence.
Related concepts
- semantic similaritymeasures · Semantic similarity is the degree to which two pieces of content or representations correspond in meaning, regardless of differences in wording or form.
- contextual embeddinguses · A contextual embedding is a numerical representation of an item whose values reflect information from its surrounding context.
- greedy matchingimplements · Greedy matching is regular-expression behavior in which a quantifier matches as much text as possible while still allowing the overall pattern to match.
- BERTuses · BERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
- evaluationused by · Evaluation is the systematic assessment of a system, model, process, or outcome against relevant criteria or objectives using evidence.
- cosine similarityuses · Cosine similarity measures the directional alignment of two nonzero vectors as the cosine of the angle between them, equal to their dot product divided by the product of their magnitudes.
Questions about BERTScore
What is BERTScore?
BERTScore is an automatic text-generation evaluation metric that compares candidate and reference texts using contextual representations of their tokens.
Which Manning books explain BERTScore?
LLM Evaluation and Alignment, The Foundational Ideas, Building Specialized AI Systems and Generative AI in Action.
What should I know before learning BERTScore?
Manning books that teach BERTScore usually introduce BLEU score, ROUGE, BERT and evaluation first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


