liveBook · Manning

Knowledge-augmented encoder

As explained in Retrieval Augmented Generation, The Foundational Ideas

A knowledge-augmented encoder is an encoder that jointly processes an input and retrieved document text to predict an output.

Retrieving a relevant document is not enough: the model must use its contents to make a better prediction. In REALM, the knowledge-augmented encoder is the reader that does this work. It takes the original input and one retrieved document, concatenates their text into a single sequence, and processes that sequence with a BERT cross-encoder. section 3.2

The knowledge retriever handles the earlier, search-oriented step. Its dual encoders embed the input and documents separately, then score them by dot product; document embeddings can be precomputed for fast search. The knowledge-augmented encoder instead processes an input and candidate document together. This separates efficient retrieval from close reading of the retrieved text. section 3.2

Figure 3.1 REALM's End-to-End Architecture. This diagram shows REALM's two-part system, which trains a retriever and a reader jointly.

The two-part diagram shows where the dual-encoder retriever ends and the knowledge-augmented reader begins during joint training. section 3.2

During masked-language-model pre-training, the reader predicts masked tokens using each candidate document. Training combines a candidate’s retrieval probability with the reader’s probability of the correct token, then sums across candidates before calculating the loss. A document becomes valuable when it helps the reader predict correctly—not merely when it looks similar to the input. This signal can refine the retriever and reader together, but it needs a warm start: REALM first trains retrieval with the Inverse Cloze Task because a randomly initialized retriever may fetch nothing useful. section 3.2 section 3.3

Three candidate documents

Suppose several retrieved documents could support a correct prediction. Training does not have to select one as the sole correct document: it combines their contributions to the prediction. In the illustrated calculation, raising the retrieval score of a helpful candidate increases its retrieval probability from 0.391 to 0.826 and lowers the loss from 0.7783 to 0.4413. The example makes the training signal visible; it is not a recipe for pre-training REALM from scratch. section 3.3

Listing 3.3 The training signal, made visible.
for bump in [0.0, 0.5, 1.0, 2.0]:  #A
boosted = retrieval_score + np.array([[bump, 0.0, 0.0]])
lp = log_softmax(boosted, axis=1)
marg = logsumexp(np.expand_dims(lp, -1) + gold_log_probs,
axis=1)
l = -(marg * mlm_mask).sum() / mlm_mask.sum()
print(f"+{bump:<4} P(z=0|x)={np.exp(lp)[0,0]:.3f} loss={l:.4f}")
 

The short score-bumping experiment isolates how changing one candidate’s retrieval score affects the joint loss. section 3.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the cross-encoder input and marginal training signal with a numerical exercise, but does not provide a from-scratch pre-training workflow.
Examples
Uses a retriever–reader diagram and a three-document score-bumping calculation to show how retrieval probability changes training loss.
What it assumes
Assumes some familiarity with dual-encoder and cross-encoder inputs while explaining the marginal loss through candidate documents.
What you can do after
Evaluate/understand it by tracing how a retrieved document helps masked-token prediction and how that performance trains retrieval.
How it connects
Connects cross-encoders to dual-encoder retrieval, BERT, masked language modeling, and marginal log-likelihood.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.