liveBook · Manning

Distillation

As explained in Retrieval Augmented Generation, The Foundational Ideas

Knowledge distillation is a training technique in which a smaller student model learns to reproduce the behavior or outputs of a larger teacher model.

Knowledge distillation trains a smaller Student model to reproduce the behavior of a larger Teacher model. Classical distillation matches the Teacher’s output probability distributions; behavioral distillation instead fine-tunes the Student with standard cross-entropy loss on Teacher-generated labeled examples. section 9.3.1

In Self-RAG, distillation transfers more than answer content: a Critic model annotates examples with reflection tokens, and the Student learns to emit those tokens while generating. This internalizes decisions about retrieval, support, and self-critique rather than implementing them as a hand-written inference loop. section 9.3 section 9.3.1

A distillation pipeline separates supervision, deployment, and runtime control:

  • Use a capable Critic or Teacher to label ordinary query-answer data with desired behaviors. section 9.3
  • Fine-tune the smaller Student with the generated text and reflection tokens; the Student need not see the Teacher’s internal logits. section 9.3.1
  • Read the Student’s learned control tokens during decoding to adjust groundedness, creativity, and retrieval decisions at runtime. section 9.3

Adaptive retrieval through distillation

A Critic annotates training examples with reflection tokens. The Student learns to predict a Retrieve token, and a probability threshold then decides whether to query the vector store or answer from parametric memory. section 9.3 section 9.3.1

Figure 9.2 The full Self-RAG lifecycle: training, decoding, and adaptive retrieval. Left: the training phase (9.3.1) routes a raw query-answer dataset through a Critic model that annotates each example with inline reflection tokens, and the resulting traces fine-tune the Student model. Middle: the inference phase (9.3.2) shows the trained Student emitting candidate continuations alongside reflection tokens, with a token-guided beam search scoring each candidate by log P(text) + Σ w_T log P(T) over the IsRel, IsSup, and IsUse tokens, with w_T exposed as runtime knobs. Right: the adaptive retrieval gate (9.3.3) compares the Retrieve-token probability to a threshold; queries above it trigger the vector store, queries below it are answered from parametric memory.

Trace the complete lifecycle from Critic-annotated data through Student fine-tuning, token-guided decoding, and thresholded retrieval. section 9.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Distinguishes probability-matching from behavioral distillation and traces annotation, fine-tuning, decoding, and retrieval gating.
Examples
Uses Critic-labeled query-answer data, reflection tokens, and Retrieve-token thresholds to demonstrate behavioral transfer.
What it assumes
Assumes familiarity with model fine-tuning, cross-entropy, retrieval, and inference-time decoding.
What you can do after
Builds a smaller model that emits learned control signals for grounded generation and adaptive retrieval.
How it connects
Connects distillation to fine-tuning, reflection tokens, self-critique, beam search, adaptive retrieval, and vector stores.

Also explained in

Compare how 14 books explain distillation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.