liveBook · Manning

Soft-target loss

As explained in Build a DeepSeek Model (From Scratch)

Soft-target loss is a divergence-based error between the probability distributions produced by a teacher and a student after temperature adjustment.

Soft-target loss measures the KL divergence between teacher and student probability distributions after dividing their logits by temperature T. It transfers the teacher’s dark knowledge: relative probabilities that encode relationships among classes. section 8.4.4

Compute teacher probabilities with softmax(teacher_logits / T), student log-probabilities with log_softmax(student_logits / T), and compare them with KL divergence. Multiplying the result by T² offsets the gradient reduction caused by temperature scaling. section 8.4.4 section 8.6.2

Use soft-target loss as part of a grounded distillation objective:

  • Combine it with hard-label cross-entropy: L = αL_hard + (1 − α)T²L_soft. section 8.4.4
  • Keep the teacher frozen and run its forward pass under torch.no_grad(); backpropagate the combined loss through the student only. section 8.6.2
  • A common weighting uses α = 0.3 for hard labels and 0.7 for soft targets, preserving ground-truth grounding while emphasizing teacher guidance. section 8.4.4

Numerical example

With teacher logits [5.0, 3.0, 1.0, −1.0], student logits [4.0, 2.5, 1.5, −0.5], and T = 3, the soft-target distributions are [0.523, 0.268, 0.138, 0.071] and [0.442, 0.268, 0.192, 0.099]. Their KL divergence is approximately 0.0199, giving L_soft ≈ 0.179 after T² scaling. With L_hard ≈ 0.275 and α = 0.3, the combined loss is approximately 0.208. section 8.6.2

L = α L_hard + β T² KL(teacher_soft || student_soft)

Shows the compact equation linking hard-label grounding, temperature-scaled soft-target KL divergence, and the combined loss. section 8.4.4

Listing 8.2 The knowledge distillation loss function.
import torch.nn.functional as F
def distillation_loss(student_logits,
     [CA] teacher_logits, labels,
     [CA] temperature=3.0, alpha=0.3):  #A
    soft_teacher = F.softmax(
     [CA] teacher_logits / temperature,
     [CA] dim=-1)  #B
    soft_student = F.log_softmax(
     [CA] student_logits / temperature,
     [CA] dim=-1)  #C
    kl_loss = F.kl_div(
     [CA] soft_student, soft_teacher,
     [CA] reduction='batchmean')  #D
 
    hard_loss = F.cross_entropy(
     [CA] student_logits, labels)  #E
    return (alpha * hard_loss
     [CA] + (1 - alpha)
     [CA] * temperature**2
     [CA] * kl_loss)  #F

Provides a PyTorch implementation using softmax, log_softmax, KL divergence, cross-entropy, and temperature scaling. section 8.4.4

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains temperature-adjusted distributions, KL divergence, T² gradient compensation, weighting, and a numerical calculation.
Examples
Uses logits, temperature-three probability vectors, KL divergence, T² scaling, weighted losses, and PyTorch implementations.
What it assumes
Assumes teacher and student logits, ground-truth labels, softmax, cross-entropy, KL divergence, and basic gradient-based training.
What you can do after
Build it with PyTorch by computing temperature-adjusted KL divergence and combining it with hard-label cross-entropy.
How it connects
Connects KL divergence, softmax, temperature scaling, cross-entropy, knowledge distillation, teacher models, and student models.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.