Logit-based distillation trains a student model to match a teacher’s probability distribution over possible next tokens rather than learning only hard one-hot labels. This transfers useful behavior into models that are cheaper, faster, and easier to deploy while preserving the teacher’s uncertainty and relationships among plausible alternatives. section 8.2.1
A practical implementation follows this sequence:
- Obtain the teacher’s token probabilities or logits for training examples. section 8.2.1
- Optionally apply temperature T to soften the distribution and expose lower-probability alternatives. section 8.2.1
- Optimize the student to match the teacher distribution, commonly with KL divergence. section 8.2.1
- Check deployment constraints: closed model APIs often expose only top-k log probabilities, truncating the long tail. section 8.2.1

