Logit-based distillation
Logit-based distillation is a knowledge-distillation method in which a student model learns to match a teacher model’s output logits or the probability distribution derived from them.
LLM Evaluation and Alignment, The Foundational Ideas
Logit-based distillation trains a student model to match a teacher model's probability distribution or logits over possible next tokens.
- How it is used
- Transfers a teacher model’s behavior into a cheaper, faster student for production deployment.
- How deep it goes
- Explains soft targets, temperature, KL divergence, and access constraints but does not provide code or detailed optimization internals.
- Examples
- Uses teacher token probabilities or logits, softened distributions, and top-k API outputs as implementation artifacts.
- What you can do after
- Build a student model that matches teacher next-token distributions using accessible logits and a distribution-matching objective.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains logit-based distillation →
Rearchitecting LLMs
Logit distillation is a knowledge-distillation method in which a student learns from a teacher model's output scores.
- How it is used
- Uses teacher output scores and hard labels to recover a pruned student model’s performance.
- How deep it goes
- Compares outcomes across three data scales and against feature-alignment variants, without detailing the logits-loss calculation.
- Examples
- Reports labels-only perplexity of 18.86, 12.65, and 11.73 for 2,000, 15,000, and 40,000 samples.
- What you can do after
- Evaluate label-only and feature-alignment distillation using perplexity, capability retention, and training-time trade-offs.
Read how Rearchitecting LLMs explains logit-based distillation →
Related concepts
- distillationkind of · Distillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model.
- FDDcontrasts with · FDD is an advanced knowledge-distillation technique that transfers feature-distribution information from a teacher model to a student model.
- Skew KLDcontrasts with · Skew KLD is a divergence-based distillation technique that compares a teacher distribution with a mixture of teacher and student distributions to smooth optimization when their predictions differ substantially.
Questions about logit-based distillation
What is logit-based distillation?
Logit-based distillation is a knowledge-distillation method in which a student model learns to match a teacher model’s output logits or the probability distribution derived from them.
Which Manning books explain logit-based distillation?
LLM Evaluation and Alignment, The Foundational Ideas and Rearchitecting LLMs.
What is the difference between logit-based distillation and sequence-level knowledge distillation?
Logit-based distillation is a knowledge-distillation method in which a student model learns to match a teacher model’s output logits or the probability distribution derived from them. Sequence-level knowledge distillation trains a student model to reproduce complete output sequences generated by a teacher model.
What is logit-based distillation a kind of?
Logit-based distillation is a kind of distillation. Distillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

