liveBook · Manning

COMET

As explained in LLM Evaluation and Alignment, The Foundational Ideas

COMET is a learned metric for evaluating machine-translation quality using multilingual representations and human-quality supervision.

A translation can preserve meaning while using different words from a reference. COMET (Crosslingual Optimized Metric for Evaluation of Translation) addresses this problem by learning to predict human quality judgments rather than counting word overlap or applying a fixed similarity formula. section 3.0 section 3.3

Unlike reference-only comparisons, the described COMET design uses three inputs: the original source, a candidate translation, and a reference translation. Considering the source helps it assess whether the candidate preserves the original meaning, including when the reference is imperfect. section 3.3.2 section 3.4.1

Evaluating a paraphrased translation

For the German source “Die Bank war voller Kunden,” a candidate is “The bank was full of customers,” while the reference is “The financial institution was crowded with clients.” COMET compares representations of all three, rather than relying only on the candidate’s word overlap with the reference. section 3.3.3

A COMET estimator processes the inputs in three steps:

  • Encode the source, candidate, and reference with a multilingual model such as XLM-RoBERTa, producing representations in a shared semantic space. section 3.3.3
  • Extract features describing how the candidate relates to the source and reference. section 3.3.3
  • Use a neural network trained on human judgments to predict a quality score. section 3.3.3

Training signals can include direct assessment ratings, human post-editing effort, or annotated errors with categories and severity. An estimator learns to predict absolute scores; a ranking model learns relative quality from better and worse candidates. section 3.3.2 section 3.3.3

Figure 3.14 Left. Estimator model architecture. The candidate, source, and reference are independently encoded using a pretrained cross-lingual encoder. The resulting embeddings are then passed through a pooling layer to create a sentence embedding for each segment. Finally, the resulting sentence embeddings are concatenated into a vector to be passed to a feed-forward regressor. Right. Ranking model architecture. It uses source, reference, a ‘better’ candidate, and a ‘worse’ candidate. They are encoded, pooled, and minimize triplet margin loss to minimize the distance between ‘better’ candidates and the ‘anchors’.

Compare how the estimator predicts a score and how the ranking model learns from translation pairs. section 3.3.3

COMET is useful when human-perceived quality and source preservation matter, especially for final model selection. Its score is not an explanation: a low value alone does not reveal whether the problem is accuracy, fluency, or style. Model bias, imperfect references, and distribution drift can also affect results, while neural inference costs more than lexical matching. Combine metrics and interpret scores in context. section 3.4.1 section 3.4.2 section 3.3.4 section 3.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains multilingual encoding, interaction features, score prediction, training signals, and estimator versus ranking variants.
Examples
Traces a German bank sentence through source, candidate, and reference representations.
What you can do after
Enables evaluation and understanding of COMET scores, including their opacity, bias, and sensitivity to drift.
How it connects
Contrasts learned quality prediction with BLEU’s lexical matching and BERTScore’s semantic similarity.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.