liveBook · Manning

Recall

As explained in LLM Customization and Fine-Tuning

A measure of how much of the probability mass or set of outcomes represented by a teacher distribution is captured by a student distribution.

Recall describes coverage, but its calculation depends on the task. In retrieval, Recall@k is the number of relevant documents retrieved divided by the total number of relevant documents. In teacher-student distillation, forward KL prioritizes distributional coverage and recall, while reverse KL favors concentration on likely outputs. section 4.5.7 section 7.5.3

Do not treat retrieval recall as interchangeable with Hit@k: Hit@k records whether any relevant document appears among the top k, so the measures agree only when each query has exactly one relevant document. section 4.5.7

Table 7.5 Divergence measures for knowledge distillation
Approach
Training signal
Student behavior
Compute cost
Tooling
Forward KL (standard SFT)
Teacher text outputs
Broad coverage, may spread thin
1x (baseline)
SFTTrainer, native
Reverse KL (on-policy)
Student generates, teacher scores
Precise, focused on the teacher’s strengths
2 to 3x
Custom training loop
Adaptive KL
Dynamic blend
Balanced coverage and precision
1.5 to 2x
Research code only
Selective token weighting
Weighted by token informativeness
Strong on distinctive patterns
1.2 to 1.5x
Custom loss function

Compare how distillation objectives trade broad coverage against focused, decisive outputs, alongside their compute costs and tooling. section 7.5.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines retrieval recall and contrasts it with Hit@k, then links distillation recall to divergence tradeoffs without deriving the objectives.
Examples
Uses a ten-query labeled set over an eight-document policy corpus and compares distillation strategies, including standard SFT.
What it assumes
Assumes labeled queries identify relevant document IDs for retrieval evaluation; distillation background is not specified.
What you can do after
Evaluate retrieval completeness against the total relevant documents and understand how distillation objectives affect coverage.
How it connects
Connects retrieval recall to precision@k and Hit@k, and distillation recall to forward KL and reverse KL.

Also explained in

Compare how 35 books explain recall →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.