liveBook · Manning

Temperature

As explained in Build a DeepSeek Model (From Scratch)

Temperature is a scalar parameter that controls the sharpness of a probability distribution produced from model logits.

Temperature T controls the sharpness of a softmax probability distribution. In classical knowledge distillation, raising T softens a teacher’s peaked output, exposing relative probabilities among alternative classes that a student can learn from. section 8.3 section 8.3.1

For logits z, compute q_i = exp(z_i / T) / Σ_j exp(z_j / T). At T=1, this is ordinary softmax. For T>1, probabilities spread more evenly while preserving their ranking. As T approaches infinity, the distribution approaches uniformity and loses its class-specific signal. A typical useful range for distillation is T=2–5, though the trade-off depends on the logits. section 8.3.1 section 8.4.3

The same logits at three temperatures

For cat, dog, fox, and bird logits [5, 3, 1, -1], T=1 produces [0.865, 0.117, 0.016, 0.002]. At T=3, divide the logits by 3 before softmax to obtain [0.523, 0.268, 0.138, 0.071]: the alternatives become more visible. At T=10, the result is [0.329, 0.270, 0.221, 0.181], nearly uniform and less informative. section 8.3.2

Figure 8.8 The same logits [5.0, 3.0, 1.0, -1.0] processed through softmax at three temperatures. At T=1, the distribution is too peaked, at T=3 the dark knowledge emerges clearly, and at T=10 the signal is lost in near-uniformity.

Compare how the same logits produce peaked, softened, and nearly uniform distributions. section 8.3.2

In distillation, apply the same temperature to teacher and student logits for the soft-target loss. Combine KL(teacher_soft || student_soft) with cross-entropy against hard labels, using the student’s unscaled logits for the latter. Multiply the KL term by T² to compensate for reduced gradient magnitude at higher temperatures: L = α L_hard + β T² KL(teacher_soft || student_soft). The hard term grounds learning in correct labels; the soft term transfers class relationships. section 8.4.4

Listing 8.2 The knowledge distillation loss function.
import torch.nn.functional as F
def distillation_loss(student_logits,
     [CA] teacher_logits, labels,
     [CA] temperature=3.0, alpha=0.3):  #A
    soft_teacher = F.softmax(
     [CA] teacher_logits / temperature,
     [CA] dim=-1)  #B
    soft_student = F.log_softmax(
     [CA] student_logits / temperature,
     [CA] dim=-1)  #C
    kl_loss = F.kl_div(
     [CA] soft_student, soft_teacher,
     [CA] reduction='batchmean')  #D
 
    hard_loss = F.cross_entropy(
     [CA] student_logits, labels)  #E
    return (alpha * hard_loss
     [CA] + (1 - alpha)
     [CA] * temperature**2
     [CA] * kl_loss)  #F

Inspect a PyTorch implementation using softmax for the teacher, log-softmax for the student, and T² scaling for the KL loss. section 8.4.4

Temperature scaling is not required for every distillation method. Supervised fine-tuning on teacher-generated reasoning traces can transfer capabilities without temperature scaling or KL divergence. section 8.9

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Works through scaled softmax numerically and implements the combined distillation loss with gradient compensation.
Examples
Compares four animal-class logits at T=1, T=3, and T=10.
What you can do after
Apply it via PyTorch to construct temperature-scaled teacher and student targets in a distillation loss.
How it connects
Connects softmax with knowledge distillation, KL divergence, and cross-entropy.

Also explained in

Compare how 20 books explain temperature →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.