Temperature T controls the sharpness of a softmax probability distribution. In classical knowledge distillation, raising T softens a teacher’s peaked output, exposing relative probabilities among alternative classes that a student can learn from. section 8.3 section 8.3.1
For logits z, compute q_i = exp(z_i / T) / Σ_j exp(z_j / T). At T=1, this is ordinary softmax. For T>1, probabilities spread more evenly while preserving their ranking. As T approaches infinity, the distribution approaches uniformity and loses its class-specific signal. A typical useful range for distillation is T=2–5, though the trade-off depends on the logits. section 8.3.1 section 8.4.3
The same logits at three temperatures
For cat, dog, fox, and bird logits [5, 3, 1, -1], T=1 produces [0.865, 0.117, 0.016, 0.002]. At T=3, divide the logits by 3 before softmax to obtain [0.523, 0.268, 0.138, 0.071]: the alternatives become more visible. At T=10, the result is [0.329, 0.270, 0.221, 0.181], nearly uniform and less informative. section 8.3.2
Figure 8.8 The same logits [5.0, 3.0, 1.0, -1.0] processed through softmax at three temperatures. At T=1, the distribution is too peaked, at T=3 the dark knowledge emerges clearly, and at T=10 the signal is lost in near-uniformity.

Compare how the same logits produce peaked, softened, and nearly uniform distributions. section 8.3.2
In distillation, apply the same temperature to teacher and student logits for the soft-target loss. Combine KL(teacher_soft || student_soft) with cross-entropy against hard labels, using the student’s unscaled logits for the latter. Multiply the KL term by T² to compensate for reduced gradient magnitude at higher temperatures: L = α L_hard + β T² KL(teacher_soft || student_soft). The hard term grounds learning in correct labels; the soft term transfers class relationships. section 8.4.4
Listing 8.2 The knowledge distillation loss function.
import torch.nn.functional as F
def distillation_loss(student_logits,
[CA] teacher_logits, labels,
[CA] temperature=3.0, alpha=0.3): #A
soft_teacher = F.softmax(
[CA] teacher_logits / temperature,
[CA] dim=-1) #B
soft_student = F.log_softmax(
[CA] student_logits / temperature,
[CA] dim=-1) #C
kl_loss = F.kl_div(
[CA] soft_student, soft_teacher,
[CA] reduction='batchmean') #D
hard_loss = F.cross_entropy(
[CA] student_logits, labels) #E
return (alpha * hard_loss
[CA] + (1 - alpha)
[CA] * temperature**2
[CA] * kl_loss) #FInspect a PyTorch implementation using softmax for the teacher, log-softmax for the student, and T² scaling for the KL loss. section 8.4.4
Temperature scaling is not required for every distillation method. Supervised fine-tuning on teacher-generated reasoning traces can transfer capabilities without temperature scaling or KL divergence. section 8.9



















