liveBook · Manning

Performance-based training

As explained in Retrieval Augmented Generation, The Foundational Ideas

A performance-based training signal is feedback derived from a system's downstream task performance and used to optimize a component.

Performance-based training optimizes a component according to its effect on downstream task performance rather than surface similarity alone. In retrieval-augmented generation, a retriever should fetch documents that help the reader answer correctly, not merely documents that resemble the query. section 3.0

REALM connects retrieval and reading through a shared training objective:

  • The dual-encoder scores candidate documents against a masked query using vector inner products. section 3.3
  • The model assigns probabilities across candidate documents and predicts masked tokens from retrieved evidence. section 3.3
  • When several documents could support a correct prediction, their probabilities are summed before applying the negative logarithm. section 3.3
  • The resulting loss sends feedback from prediction quality back to retrieval, teaching the retriever to prefer useful evidence. section 3.3 section 3.5

Making the signal visible

A three-document calculation starts with retrieval probabilities of 0.391, 0.233, and 0.376 and a loss of 0.7783. Increasing the first document's retrieval score raises its probability to 0.826 and lowers the loss to 0.4413, showing how retrieval preference follows downstream prediction value.

# [batch_size, num_candidates]
candidate_score = retrieval_score
candidate_log_probs = tf.math.log_softmax(candidate_score)
…
# [batch_size, num_candidates, num_masks]
joint_gold_log_probs = (
    tf.expand_dims(candidate_log_probs, -1) + gold_log_probs
)
# [batch_size, num_masks]
marginal_gold_log_probs = tf.reduce_logsumexp(joint_gold_log_probs, 1)

Inspect the tensor operations that combine candidate log-probabilities with answer probabilities and marginalize over sufficient documents. section 3.3

The approach is self-supervised: masked tokens provide targets without external labels, while the retriever learns from the reader's prediction objective. Its cost is substantial end-to-end pretraining, including asynchronous index refreshes, which can make the method impractical for many applications. section 3.5

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the retriever-reader loss, marginalization, code-level tensors, and score perturbations, but stops short of runnable pretraining.
Examples
Uses a three-document NumPy marginalization and controlled retrieval-score boosts to expose probability and loss changes.
What it assumes
Assumes vector representations and probability basics while introducing masked-token targets and marginal log-likelihood through examples.
What you can do after
Evaluate retrieval usefulness through marginal log-likelihood and understand how reader performance updates retriever preferences.
How it connects
Connects information retrieval, dual encoders, cross encoders, masked language modeling, self-supervised learning, and retrieval-augmented generation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.