liveBook · Manning

Perplexity distillation

As explained in Retrieval Augmented Generation, The Foundational Ideas

Perplexity distillation is a training objective that converts reader-derived document utility scores into a target distribution and trains the retriever to match it.

A retriever can rank a passage that repeats a question above one that answers it. Perplexity distillation (PDist) uses the reader to correct that ranking: documents receive higher training weight when they make the target answer more likely. It needs question–answer examples, but no document-relevance labels. section 6.2.2 section 6.7

For each training question and target answer:

  • Retrieve the top-k documents. Give each document to the reader separately and measure the log-probability it assigns to the target answer. section 6.2.2 section 6.2.3
  • Apply softmax to those scores to form a target distribution. Compare it with the retriever’s distribution over the same documents using KL divergence, then update the retriever. section 6.2.3

Moving an explanation above a lexical match

Asked why AWS operating margins declined, a similarity-based retriever initially puts 0.989 probability on a passage quoting margin figures and just 0.004 on the passage explaining the decline. The reader assigns the explanatory passage a target-answer log-probability of −1.3, versus roughly −35 to −38 for the others. In a laptop demonstration, 20 retriever updates shift its probability to 0.999 on the explanation. The reader is frozen in this demonstration; its scores supply the supervision. section 6.2.3 section 6.8

Listing 6.3 One step of Perplexity Distillation.
retriever = SentenceTransformer("all-MiniLM-L6-v2")
opt = torch.optim.Adam(retriever.parameters(), lr=2e-5)
 
def retriever_dist() -> torch.Tensor:
    """Question-to-passage similarity, as logits."""
    feats = retriever.tokenize([QUESTION] + PASSAGES)
    emb = F.normalize(  #A
        retriever(feats)["sentence_embedding"], dim=-1)
    return emb[0] @ emb[1:].T / 0.1  #B
for step in range(20):
    loss = F.kl_div(  #C
        F.log_softmax(retriever_dist(), dim=0),
        target, reduction="sum")
    opt.zero_grad()
    loss.backward()  #D
    opt.step()
 
 

The training step shows the retriever’s softmax scores being matched to the reader-derived target with KL divergence. section 6.2.3

PDist is a relatively cheap feedback signal, not a substitute for a useful starting retriever: a random retriever may fetch no helpful document, leaving the reader unable to distinguish candidates. In Atlas’s 64-shot ablation, PDist scores 45.7 against 42.2 for a fixed retriever; omitting joint pre-training drops the score to 30.0. Other tested losses perform similarly, while leave-one-out evaluation more closely reflects multi-document inference at higher computational cost. For a full pipeline, start from a jointly pre-trained retriever–reader checkpoint rather than treating the small demonstration as a deployment recipe. section 6.2.2 section 6.2.3 section 6.7

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Works through per-document reader scores, softmax targets, KL loss, and a small training run; defers the full pre-training pipeline.
Examples
Uses an operating-margin question, four passages, reader log-probabilities, and a 20-step retriever update.
What it assumes
Assumes question–answer examples and a capable reader; requires an already trained retriever to fetch useful candidates, but no passage-relevance labels.
What you can do after
Build it from scratch as a small reader-supervised retriever update, or apply it through a jointly pre-trained Atlas pipeline.
How it connects
Connects retrieval-augmented generation, reader–retriever joint training, few-shot fine-tuning, and alternative retriever losses.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.