Retrieving a relevant document is not enough: the model must use its contents to make a better prediction. In REALM, the knowledge-augmented encoder is the reader that does this work. It takes the original input and one retrieved document, concatenates their text into a single sequence, and processes that sequence with a BERT cross-encoder. section 3.2
The knowledge retriever handles the earlier, search-oriented step. Its dual encoders embed the input and documents separately, then score them by dot product; document embeddings can be precomputed for fast search. The knowledge-augmented encoder instead processes an input and candidate document together. This separates efficient retrieval from close reading of the retrieved text. section 3.2
Figure 3.1 REALM's End-to-End Architecture. This diagram shows REALM's two-part system, which trains a retriever and a reader jointly.

The two-part diagram shows where the dual-encoder retriever ends and the knowledge-augmented reader begins during joint training. section 3.2
During masked-language-model pre-training, the reader predicts masked tokens using each candidate document. Training combines a candidate’s retrieval probability with the reader’s probability of the correct token, then sums across candidates before calculating the loss. A document becomes valuable when it helps the reader predict correctly—not merely when it looks similar to the input. This signal can refine the retriever and reader together, but it needs a warm start: REALM first trains retrieval with the Inverse Cloze Task because a randomly initialized retriever may fetch nothing useful. section 3.2 section 3.3
Three candidate documents
Suppose several retrieved documents could support a correct prediction. Training does not have to select one as the sole correct document: it combines their contributions to the prediction. In the illustrated calculation, raising the retrieval score of a helpful candidate increases its retrieval probability from 0.391 to 0.826 and lowers the loss from 0.7783 to 0.4413. The example makes the training signal visible; it is not a recipe for pre-training REALM from scratch. section 3.3
Listing 3.3 The training signal, made visible.
for bump in [0.0, 0.5, 1.0, 2.0]: #A
boosted = retrieval_score + np.array([[bump, 0.0, 0.0]])
lp = log_softmax(boosted, axis=1)
marg = logsumexp(np.expand_dims(lp, -1) + gold_log_probs,
axis=1)
l = -(marg * mlm_mask).sum() / mlm_mask.sum()
print(f"+{bump:<4} P(z=0|x)={np.exp(lp)[0,0]:.3f} loss={l:.4f}")
The short score-bumping experiment isolates how changing one candidate’s retrieval score affects the joint loss. section 3.3
