liveBook · Manning

Logit-based distillation

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Logit-based distillation trains a student model to match a teacher model's probability distribution or logits over possible next tokens.

Logit-based distillation trains a student model to match a teacher’s probability distribution over possible next tokens rather than learning only hard one-hot labels. This transfers useful behavior into models that are cheaper, faster, and easier to deploy while preserving the teacher’s uncertainty and relationships among plausible alternatives. section 8.2.1

A practical implementation follows this sequence:

  • Obtain the teacher’s token probabilities or logits for training examples. section 8.2.1
  • Optionally apply temperature T to soften the distribution and expose lower-probability alternatives. section 8.2.1
  • Optimize the student to match the teacher distribution, commonly with KL divergence. section 8.2.1
  • Check deployment constraints: closed model APIs often expose only top-k log probabilities, truncating the long tail. section 8.2.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains soft targets, temperature, KL divergence, and access constraints but does not provide code or detailed optimization internals.
Examples
Uses teacher token probabilities or logits, softened distributions, and top-k API outputs as implementation artifacts.
What it assumes
Assumes basic supervised learning while clarifying why probability distributions provide more information than one-hot labels.
What you can do after
Build a student model that matches teacher next-token distributions using accessible logits and a distribution-matching objective.
How it connects
Connects supervised learning, KL divergence, rationale distillation, and judge-filtered distillation.

Also explained in

Compare how 2 books explain logit-based distillation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.