Recall describes coverage, but its calculation depends on the task. In retrieval, Recall@k is the number of relevant documents retrieved divided by the total number of relevant documents. In teacher-student distillation, forward KL prioritizes distributional coverage and recall, while reverse KL favors concentration on likely outputs. section 4.5.7 section 7.5.3
Do not treat retrieval recall as interchangeable with Hit@k: Hit@k records whether any relevant document appears among the top k, so the measures agree only when each query has exactly one relevant document. section 4.5.7
Table 7.5 Divergence measures for knowledge distillation
|
Approach
|
Training signal
|
Student behavior
|
Compute cost
|
Tooling
|
|
Forward KL (standard SFT)
|
Teacher text outputs
|
Broad coverage, may spread
thin
|
1x (baseline)
|
SFTTrainer,
native
|
|
Reverse KL (on-policy)
|
Student generates, teacher
scores
|
Precise, focused on the
teacher’s strengths
|
2 to 3x
|
Custom training loop
|
|
Adaptive KL
|
Dynamic blend
|
Balanced coverage and precision
|
1.5 to 2x
|
Research code only
|
|
Selective token weighting
|
Weighted by token informativeness
|
Strong on distinctive patterns
|
1.2 to 1.5x
|
Custom loss function
|
Compare how distillation objectives trade broad coverage against focused, decisive outputs, alongside their compute costs and tooling. section 7.5.3


































