BERTScore is a text-generation evaluation metric that compares candidate and reference texts using contextual token representations from BERT-like models.
BERTScore evaluates generated text by comparing candidate and reference meanings through contextualized token embeddings from pretrained language models such as BERT. section 1.4.2
Its implementation-oriented evaluation uses:
Token-level precision, recall, and F1 scores based on contextual representations. section 1.4.2
Semantic comparison that accommodates multiple valid expressions of the same concept, unlike exact token-overlap metrics. section 1.4.2
Human expert review and domain-specific checks alongside automated scoring, especially for professional or production applications. section 1.4.2
How Building Specialized AI Systems covers it
How deep it goes
Explains contextual embeddings and token-level scoring, but defers model-specific implementation details and metric limitations beyond supplementation.
Examples
Uses generated and reference texts, with legal-document and customer-service evaluation needs as domain examples.
What it assumes
Builds on token-similarity metrics and pretrained language-model representations while treating their detailed mechanics as background.
What you can do after
Evaluate text-generation quality with semantic similarity scores while supplementing them with expert and application-specific checks.
How it connects
Contrasts with BLEU, ROUGE, and METEOR, and complements human evaluation and custom domain-specific metrics.
Generative AI in ActionBERTScore is an automatic metric that evaluates generated text by comparing contextual semantic representations of candidate and reference text.
semantic similaritymeasures · Semantic similarity is the degree to which two pieces of content or representations correspond in meaning, regardless of differences in wording or form.
contextual embeddinguses · A contextual embedding is a numerical representation of an item whose values reflect information from its surrounding context.
greedy matchingimplements · Greedy matching is regular-expression behavior in which a quantifier matches as much text as possible while still allowing the overall pattern to match.
BERTuses · BERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
evaluationused by · Evaluation is the systematic assessment of a system, model, process, or outcome against relevant criteria or objectives using evidence.
cosine similarityuses · Cosine similarity measures the directional alignment of two nonzero vectors as the cosine of the angle between them, equal to their dot product divided by the product of their magnitudes.
This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.