A semantic similarity metric evaluates generated text by comparing meaning and context rather than exact word overlap. It is useful when multiple phrasings can correctly express the same concept. section 1.4.2
BERTScore is a semantic similarity metric that uses contextualized embeddings from pretrained language models such as BERT. It computes token-level precision, recall, and F1, providing a more nuanced quality signal than overlap-only measures. section 1.4.2
Use semantic similarity metrics alongside broader evaluation because text quality also depends on application-specific requirements:
- Contrast them with BLEU, ROUGE, and METEOR, which assess token overlap and have known limitations. section 1.4.2
- Supplement automated scores with domain-expert review for professional standards and organizational requirements. section 1.4.2
- Add custom criteria such as citation accuracy, formatting, helpfulness, brand voice, or policy accuracy when the application demands them. section 1.4.2
