liveBook · Manning

Joint probability distribution

As explained in Retrieval Augmented Generation, The Foundational Ideas

Joint probability is the probability that multiple events or variables take specified values together.

A joint probability measures the chance that multiple events occur together. In REALM, it represents both retrieving document z and producing the masked answer y given query x: P(y, z | x). section 3.3

REALM combines probabilities in three steps:

  • Estimate P(z | x) from retrieval scores, using a softmax over candidate documents. section 3.3
  • Estimate P(y | x, z), the masked-token probability conditioned on each retrieved document. section 3.3
  • Add their log probabilities to obtain the joint log probability, then sum across documents with log-sum-exp to obtain P(y | x). section 3.3

Three-document calculation

A NumPy example computes document scores, converts them to P(z | x), combines them with masked-token log probabilities, and reports marginal probabilities of 0.356 and 0.593. Increasing one document's retrieval score raises its probability and lowers the loss, making the training signal visible. section 3.3

# [batch_size, num_candidates]
candidate_score = retrieval_score
candidate_log_probs = tf.math.log_softmax(candidate_score)
…
# [batch_size, num_candidates, num_masks]
joint_gold_log_probs = (
    tf.expand_dims(candidate_log_probs, -1) + gold_log_probs
)
# [batch_size, num_masks]
marginal_gold_log_probs = tf.reduce_logsumexp(joint_gold_log_probs, 1)

Inspect the tensor operations that combine candidate log probabilities with answer log probabilities and marginalize over documents. section 3.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains definition, log-space computation, marginalization, tensor shapes, and a small numerical example.
Examples
Demonstrates three candidate documents, NumPy log-sum-exp marginalization, and retrieval-score boosts that reduce loss.
What it assumes
Assumes basic probability and tensor operations while defining conditional, marginal, and joint probability distinctions explicitly.
What you can do after
Evaluate joint and marginal probabilities and implement their log-space calculation for document retrieval training.
How it connects
Connects joint probability to conditional probability, marginal probability, marginalization, maximum likelihood, and retrieval.

Also explained in

Compare how 6 books explain joint probability distribution →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.