Dark knowledge is the information in a teacher model’s output probabilities beyond its most likely prediction. The relative probabilities of alternative classes reveal relationships that a hard label discards. In knowledge distillation, these soft targets give a student a richer signal than the correct answer alone. section 8.2.1 section 8.9
A standard softmax can hide those relationships in tiny probabilities. Temperature scaling divides the logits by T before softmax: T > 1 spreads probability across alternatives while preserving their ranking. Too much smoothing makes the distribution nearly uniform and dilutes the useful signal. A typical starting range is T=2–5; smaller students may benefit from the lower end. section 8.3 section 8.3.5
Revealing alternatives
For cat, dog, fox, and bird logits [5, 3, 1, -1], softmax at T=1 gives approximately [0.865, 0.117, 0.016, 0.002]. At T=3 it gives [0.523, 0.268, 0.138, 0.071]: cat remains the top prediction, but the relative plausibility of dog, fox, and bird is clearer. At T=10, the distribution is nearly uniform. section 8.3.2
Figure 8.8 The same logits [5.0, 3.0, 1.0, -1.0] processed through softmax at three temperatures. At T=1, the distribution is too peaked, at T=3 the dark knowledge emerges clearly, and at T=10 the signal is lost in near-uniformity.

Compare the same logits at three temperatures to see the trade-off between a peaked and a nearly uniform distribution. section 8.3.2
To transfer dark knowledge, match teacher and student distributions at the same temperature using KL divergence. Combine that soft-target loss with cross-entropy against hard labels: L = α L_hard + β T² KL(teacher_soft || student_soft). The T² factor compensates for reduced soft-loss gradients at higher temperatures; the weights control reliance on ground truth versus the teacher. section 8.4.4 section 8.3.4
Listing 8.2 The knowledge distillation loss function.
import torch.nn.functional as F
def distillation_loss(student_logits,
[CA] teacher_logits, labels,
[CA] temperature=3.0, alpha=0.3): #A
soft_teacher = F.softmax(
[CA] teacher_logits / temperature,
[CA] dim=-1) #B
soft_student = F.log_softmax(
[CA] student_logits / temperature,
[CA] dim=-1) #C
kl_loss = F.kl_div(
[CA] soft_student, soft_teacher,
[CA] reduction='batchmean') #D
hard_loss = F.cross_entropy(
[CA] student_logits, labels) #E
return (alpha * hard_loss
[CA] + (1 - alpha)
[CA] * temperature**2
[CA] * kl_loss) #FThe PyTorch loss computes teacher softmax, student log-softmax, KL divergence, and hard-label cross-entropy before combining them. section 8.4.4
Distillation need not always transfer dark knowledge through probabilities. An alternative for reasoning models is to fine-tune a student on teacher-generated reasoning traces rather than match temperature-scaled logits. section 8.5.1
