Suppose several retrieved passages could help predict a masked word. Choosing one passage as the only correct support would discard useful evidence from the others. Marginal likelihood instead measures the probability of the word after accounting for the possible passages. section 3.3
In REALM, the retrieved document z is a latent variable. For an input x and correct word y, the model sums p(z | x) × p(y | z, x) over documents z. The first term is the retriever’s probability of selecting a document; the second is the encoder’s probability of predicting the word when given that document. Training maximizes the log of this sum, linking retriever and encoder performance. section 3.2
What a better retrieval score changes
In a three-document NumPy calculation, increasing the score of a useful document raises its retrieval probability from 0.391 to 0.826. The resulting masked-word loss falls from 0.7783 to 0.4413. The retriever is rewarded because putting more weight on that document improves the prediction—not merely because its embedding looks similar to the query. section 3.3
# [batch_size, num_candidates]
candidate_score = retrieval_score
candidate_log_probs = tf.math.log_softmax(candidate_score)
…
# [batch_size, num_candidates, num_masks]
joint_gold_log_probs = (
tf.expand_dims(candidate_log_probs, -1) + gold_log_probs
)
# [batch_size, num_masks]
marginal_gold_log_probs = tf.reduce_logsumexp(joint_gold_log_probs, 1)The TensorFlow operations add each candidate’s retrieval log-probability to its correct-word log-probability, then use log-sum-exp to marginalize across candidates. section 3.3
Summing over an entire corpus is not computationally practical, so REALM uses the retriever’s top-k candidates. A relevant passage can lift the marginal probability of a rare answer even when that passage has a modest retrieval probability. The practical constraint is that this approximation depends on which documents make the candidate set. section 3.2 section 3.4


