liveBook · Manning

Learned metric

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A meta-learned metric is an evaluation measure whose judging behavior is optimized to reproduce the judgments of another evaluator, such as humans.

A learned metric predicts evaluation judgments using a model trained on examples of those judgments. It is useful when good answers can be worded differently, so word overlap alone is an unreliable measure of quality. section 1.1 section 3.1.1

A designed metric applies an explicit formula; a learned metric learns a relationship between inputs and human quality assessments. Semantic similarity is still a proxy: a translation can resemble a reference yet contain an error apparent to a human. section 3.3.1

COMET for translation

COMET fine-tunes a pretrained language model on human translation assessments. It evaluates a source text, a candidate translation, and a reference translation together, helping it detect meaning changes that a reference-only comparison can miss. Training targets can include quality ratings, post-editing effort, or pairwise comparisons. section 3.3 section 3.3.1 section 3.3.4

A prompted LLM judge is not automatically a learned metric: using a model to judge is distinct from aligning its judgments with human assessments. Tuning judging instructions to correlate with human evaluators can instead be framed as a meta-learned metric. section 3.1.1 section 4.2

Learned scores are proxies, not explanations. COMET requires human-judged training data, can degrade under distribution drift, and is slower and less interpretable than overlap metrics. Combine measures and choose them for the task: fast lexical metrics can support iteration, while more expensive measures can help with final selection. section 3.3.4 section 3.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains how COMET learns from human assessments and uses source, candidate, and reference inputs, while noting its scores lack explanations.
Examples
Uses COMET to evaluate translations and contrasts its learned judgments with BERTScore similarity.
What you can do after
Apply it via the unbabel-comet Python package to score translation batches and weigh results alongside other metrics.
How it connects
Connects learned evaluation to human judgments, lexical metrics, semantic similarity, and LLM-as-a-judge.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.