A translation can preserve meaning while using different words from a reference. COMET (Crosslingual Optimized Metric for Evaluation of Translation) addresses this problem by learning to predict human quality judgments rather than counting word overlap or applying a fixed similarity formula. section 3.0 section 3.3
Unlike reference-only comparisons, the described COMET design uses three inputs: the original source, a candidate translation, and a reference translation. Considering the source helps it assess whether the candidate preserves the original meaning, including when the reference is imperfect. section 3.3.2 section 3.4.1
Evaluating a paraphrased translation
For the German source “Die Bank war voller Kunden,” a candidate is “The bank was full of customers,” while the reference is “The financial institution was crowded with clients.” COMET compares representations of all three, rather than relying only on the candidate’s word overlap with the reference. section 3.3.3
A COMET estimator processes the inputs in three steps:
- Encode the source, candidate, and reference with a multilingual model such as XLM-RoBERTa, producing representations in a shared semantic space. section 3.3.3
- Extract features describing how the candidate relates to the source and reference. section 3.3.3
- Use a neural network trained on human judgments to predict a quality score. section 3.3.3
Training signals can include direct assessment ratings, human post-editing effort, or annotated errors with categories and severity. An estimator learns to predict absolute scores; a ranking model learns relative quality from better and worse candidates. section 3.3.2 section 3.3.3
Figure 3.14 Left. Estimator model architecture. The candidate, source, and reference are independently encoded using a pretrained cross-lingual encoder. The resulting embeddings are then passed through a pooling layer to create a sentence embedding for each segment. Finally, the resulting sentence embeddings are concatenated into a vector to be passed to a feed-forward regressor. Right. Ranking model architecture. It uses source, reference, a ‘better’ candidate, and a ‘worse’ candidate. They are encoded, pooled, and minimize triplet margin loss to minimize the distance between ‘better’ candidates and the ‘anchors’.
Compare how the estimator predicts a score and how the ranking model learns from translation pairs. section 3.3.3
COMET is useful when human-perceived quality and source preservation matter, especially for final model selection. Its score is not an explanation: a low value alone does not reveal whether the problem is accuracy, fluency, or style. Model bias, imperfect references, and distribution drift can also affect results, while neural inference costs more than lexical matching. Combine metrics and interpret scores in context. section 3.4.1 section 3.4.2 section 3.3.4 section 3.5
