A learned metric predicts evaluation judgments using a model trained on examples of those judgments. It is useful when good answers can be worded differently, so word overlap alone is an unreliable measure of quality. section 1.1 section 3.1.1
A designed metric applies an explicit formula; a learned metric learns a relationship between inputs and human quality assessments. Semantic similarity is still a proxy: a translation can resemble a reference yet contain an error apparent to a human. section 3.3.1
COMET for translation
COMET fine-tunes a pretrained language model on human translation assessments. It evaluates a source text, a candidate translation, and a reference translation together, helping it detect meaning changes that a reference-only comparison can miss. Training targets can include quality ratings, post-editing effort, or pairwise comparisons. section 3.3 section 3.3.1 section 3.3.4
A prompted LLM judge is not automatically a learned metric: using a model to judge is distinct from aligning its judgments with human assessments. Tuning judging instructions to correlate with human evaluators can instead be framed as a meta-learned metric. section 3.1.1 section 4.2
Learned scores are proxies, not explanations. COMET requires human-judged training data, can degrade under distribution drift, and is slower and less interpretable than overlap metrics. Combine measures and choose them for the task: fast lexical metrics can support iteration, while more expensive measures can help with final selection. section 3.3.4 section 3.5
