liveBook · Manning

Cross-encoder

As explained in Retrieval Augmented Generation, The Foundational Ideas

A cross-encoder jointly processes two input sequences through one encoder so that tokens from both sequences can interact directly during encoding.

A cross-encoder scores two texts by processing them together, rather than comparing representations made separately. In a BERT-style model, the texts form one packed input, and self-attention lets tokens in either text interact with tokens in the other. A prediction, such as a similarity score, can be made from the final [CLS] representation. section 2.3

Why not score every document?

To route a new support ticket, a cross-encoder could compare it with each of 100,000 resolved tickets. Each pair needs its own model pass; the candidate tickets cannot be encoded once and reused. At the measured rate cited for this scenario, routing one ticket would take about eight minutes of GPU time. section 2.3

Use the expensive joint reading where it earns its cost:

  • Retrieve candidates with a bi-encoder, whose document vectors can be prepared and indexed in advance; then cross-encode only a small candidate set to rerank it. section 7.6.3 section 8.6
  • In REALM, retrieve documents with a dual-encoder, then jointly process the input and a retrieved document with a knowledge-augmented cross-encoder to predict a masked token. section 3.2
  • Check rankings against the deployment corpus. An off-the-shelf cross-encoder can rank an outdated policy above the directive replacing it; a strong relevance score does not establish that a source is current. section 8.6

A packed sentence pair

For “The cat sat on the mat” and “A feline rested on the rug,” a BERT-style input is `[CLS] The cat sat on the mat. [SEP] A feline rested on the rug. [SEP]`. The separators mark the sentence boundaries; [CLS] supplies a representation for a final pair-level prediction. section 2.3

Figure 3.1 REALM's End-to-End Architecture. This diagram shows REALM's two-part system, which trains a retriever and a reader jointly.

The REALM architecture diagram shows the division of labor: an indexed dual-encoder finds knowledge, and a cross-encoder jointly reads the input and retrieved document. section 3.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains packed BERT inputs, token interaction, pair-level prediction, and per-pair inference cost without implementing a cross-encoder.
Examples
Uses support-ticket matching, a packed sentence pair, REALM's retriever–reader architecture, and a retrieve-and-rerank workflow.
What it assumes
Builds from a sentence-pair example and explains [CLS] and [SEP], while relying on basic familiarity with transformer models.
What you can do after
Apply it via a reranker after initial retrieval, limiting scored candidates and checking rankings for corpus-specific errors.
How it connects
Connects self-attention and BERT with bi-encoder retrieval, reranking, and retrieval-augmented generation.

Also explained in

Compare how 4 books explain cross-encoder →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.