liveBook · Manning

Logit-based distillation

Logit-based distillation is a knowledge-distillation method in which a student model learns to match a teacher model’s output logits or the probability distribution derived from them.

How 2 Manning books explain it

LLM Evaluation and Alignment, The Foundational Ideas

Logit-based distillation trains a student model to match a teacher model's probability distribution or logits over possible next tokens.

How it is used
Transfers a teacher model’s behavior into a cheaper, faster student for production deployment.
How deep it goes
Explains soft targets, temperature, KL divergence, and access constraints but does not provide code or detailed optimization internals.
Examples
Uses teacher token probabilities or logits, softened distributions, and top-k API outputs as implementation artifacts.
What you can do after
Build a student model that matches teacher next-token distributions using accessible logits and a distribution-matching objective.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains logit-based distillation →

Rearchitecting LLMs

Logit distillation is a knowledge-distillation method in which a student learns from a teacher model's output scores.

How it is used
Uses teacher output scores and hard labels to recover a pruned student model’s performance.
How deep it goes
Compares outcomes across three data scales and against feature-alignment variants, without detailing the logits-loss calculation.
Examples
Reports labels-only perplexity of 18.86, 12.65, and 11.73 for 2,000, 15,000, and 40,000 samples.
What you can do after
Evaluate label-only and feature-alignment distillation using perplexity, capability retention, and training-time trade-offs.

Read how Rearchitecting LLMs explains logit-based distillation →

Related concepts

Questions about logit-based distillation

What is logit-based distillation?

Logit-based distillation is a knowledge-distillation method in which a student model learns to match a teacher model’s output logits or the probability distribution derived from them.

Which Manning books explain logit-based distillation?

LLM Evaluation and Alignment, The Foundational Ideas and Rearchitecting LLMs.

What is the difference between logit-based distillation and sequence-level knowledge distillation?

Logit-based distillation is a knowledge-distillation method in which a student model learns to match a teacher model’s output logits or the probability distribution derived from them. Sequence-level knowledge distillation trains a student model to reproduce complete output sequences generated by a teacher model.

What is logit-based distillation a kind of?

Logit-based distillation is a kind of distillation. Distillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.