A joint probability measures the chance that multiple events occur together. In REALM, it represents both retrieving document z and producing the masked answer y given query x: P(y, z | x). section 3.3
REALM combines probabilities in three steps:
- Estimate P(z | x) from retrieval scores, using a softmax over candidate documents. section 3.3
- Estimate P(y | x, z), the masked-token probability conditioned on each retrieved document. section 3.3
- Add their log probabilities to obtain the joint log probability, then sum across documents with log-sum-exp to obtain P(y | x). section 3.3
Three-document calculation
A NumPy example computes document scores, converts them to P(z | x), combines them with masked-token log probabilities, and reports marginal probabilities of 0.356 and 0.593. Increasing one document's retrieval score raises its probability and lowers the loss, making the training signal visible. section 3.3
# [batch_size, num_candidates]
candidate_score = retrieval_score
candidate_log_probs = tf.math.log_softmax(candidate_score)
…
# [batch_size, num_candidates, num_masks]
joint_gold_log_probs = (
tf.expand_dims(candidate_log_probs, -1) + gold_log_probs
)
# [batch_size, num_masks]
marginal_gold_log_probs = tf.reduce_logsumexp(joint_gold_log_probs, 1)Inspect the tensor operations that combine candidate log probabilities with answer log probabilities and marginalize over documents. section 3.3





