liveBook · Manning

BERTScore

As explained in Building Specialized AI Systems

BERTScore is a text-generation evaluation metric that compares candidate and reference texts using contextual token representations from BERT-like models.

BERTScore evaluates generated text by comparing candidate and reference meanings through contextualized token embeddings from pretrained language models such as BERT. section 1.4.2

Its implementation-oriented evaluation uses:

  • Token-level precision, recall, and F1 scores based on contextual representations. section 1.4.2
  • Semantic comparison that accommodates multiple valid expressions of the same concept, unlike exact token-overlap metrics. section 1.4.2
  • Human expert review and domain-specific checks alongside automated scoring, especially for professional or production applications. section 1.4.2

How Building Specialized AI Systems covers it

How deep it goes
Explains contextual embeddings and token-level scoring, but defers model-specific implementation details and metric limitations beyond supplementation.
Examples
Uses generated and reference texts, with legal-document and customer-service evaluation needs as domain examples.
What it assumes
Builds on token-similarity metrics and pretrained language-model representations while treating their detailed mechanics as background.
What you can do after
Evaluate text-generation quality with semantic similarity scores while supplementing them with expert and application-specific checks.
How it connects
Contrasts with BLEU, ROUGE, and METEOR, and complements human evaluation and custom domain-specific metrics.

Also explained in

Compare how 3 books explain BERTScore →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.