Knowledge distillation trains a smaller Student model to reproduce the behavior of a larger Teacher model. Classical distillation matches the Teacher’s output probability distributions; behavioral distillation instead fine-tunes the Student with standard cross-entropy loss on Teacher-generated labeled examples. section 9.3.1
In Self-RAG, distillation transfers more than answer content: a Critic model annotates examples with reflection tokens, and the Student learns to emit those tokens while generating. This internalizes decisions about retrieval, support, and self-critique rather than implementing them as a hand-written inference loop. section 9.3 section 9.3.1
A distillation pipeline separates supervision, deployment, and runtime control:
- Use a capable Critic or Teacher to label ordinary query-answer data with desired behaviors. section 9.3
- Fine-tune the smaller Student with the generated text and reflection tokens; the Student need not see the Teacher’s internal logits. section 9.3.1
- Read the Student’s learned control tokens during decoding to adjust groundedness, creativity, and retrieval decisions at runtime. section 9.3
Adaptive retrieval through distillation
A Critic annotates training examples with reflection tokens. The Student learns to predict a Retrieve token, and a probability threshold then decides whether to query the vector store or answer from parametric memory. section 9.3 section 9.3.1
Figure 9.2 The full Self-RAG lifecycle: training, decoding, and adaptive retrieval. Left: the training phase (9.3.1) routes a raw query-answer dataset through a Critic model that annotates each example with inline reflection tokens, and the resulting traces fine-tune the Student model. Middle: the inference phase (9.3.2) shows the trained Student emitting candidate continuations alongside reflection tokens, with a token-guided beam search scoring each candidate by log P(text) + Σ w_T log P(T) over the IsRel, IsSup, and IsUse tokens, with w_T exposed as runtime knobs. Right: the adaptive retrieval gate (9.3.3) compares the Retrieve-token probability to a threshold; queries above it trigger the vector store, queries below it are answered from parametric memory.

Trace the complete lifecycle from Critic-annotated data through Student fine-tuning, token-guided decoding, and thresholded retrieval. section 9.3













