liveBook · Manning

Semantic similarity metric

As explained in Building Specialized AI Systems

A metric that evaluates similarity between texts based on meaning and context rather than exact word overlap.

A semantic similarity metric evaluates generated text by comparing meaning and context rather than exact word overlap. It is useful when multiple phrasings can correctly express the same concept. section 1.4.2

BERTScore is a semantic similarity metric that uses contextualized embeddings from pretrained language models such as BERT. It computes token-level precision, recall, and F1, providing a more nuanced quality signal than overlap-only measures. section 1.4.2

Use semantic similarity metrics alongside broader evaluation because text quality also depends on application-specific requirements:

  • Contrast them with BLEU, ROUGE, and METEOR, which assess token overlap and have known limitations. section 1.4.2
  • Supplement automated scores with domain-expert review for professional standards and organizational requirements. section 1.4.2
  • Add custom criteria such as citation accuracy, formatting, helpfulness, brand voice, or policy accuracy when the application demands them. section 1.4.2

How Building Specialized AI Systems covers it

How deep it goes
Explains BERTScore through contextualized embeddings and token-level precision, recall, and F1.
Examples
Uses legal document generation and customer-service responses to illustrate custom evaluation criteria.
What you can do after
Evaluate/understand when semantic comparison should supplement token-overlap metrics and expert review.
How it connects
Contrasts BERTScore with BLEU, ROUGE, METEOR, human evaluation, and domain-specific metrics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.