liveBook · Manning

Cross-lingual encoder

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A cross-lingual encoder is a text encoder that maps multiple human languages into representations suitable for direct comparison.

A cross-lingual encoder maps text from multiple human languages into a shared embedding space, enabling direct comparison across languages. COMET uses XLM-RoBERTa to independently encode the source, candidate, and reference, producing contextual embeddings for each token. section 3.3.3

Within COMET’s architecture, the encoder:

  • Processes source, candidate, and reference inputs separately. section 3.3.3
  • Produces contextual token embeddings for each input. section 3.3.3
  • Feeds multilingual representations into pooling and task-specific prediction components. section 3.3.3
Figure 3.14 Left. Estimator model architecture. The candidate, source, and reference are independently encoded using a pretrained cross-lingual encoder. The resulting embeddings are then passed through a pooling layer to create a sentence embedding for each segment. Finally, the resulting sentence embeddings are concatenated into a vector to be passed to a feed-forward regressor. Right. Ranking model architecture. It uses source, reference, a ‘better’ candidate, and a ‘worse’ candidate. They are encoded, pooled, and minimize triplet margin loss to minimize the distance between ‘better’ candidates and the ‘anchors’.

The architecture figure shows how independently encoded source, candidate, and reference representations become sentence embeddings before regression or ranking. section 3.3.3

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Descends from multilingual token embeddings to shared representations, then defers sentence compression to pooling layers.
Examples
Uses source, candidate, and reference translations encoded independently for estimator and ranking architectures.
What you can do after
Evaluate multilingual translation quality by producing comparable representations for source, candidate, and reference text.
How it connects
Connects XLM-RoBERTa, transformer models, pooling, sentence embeddings, COMET, BERTScore, and BLEU.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.