liveBook · Manning

Hard-label loss

As explained in Build a DeepSeek Model (From Scratch)

Hard-label loss is the cross-entropy error between a student's predicted token or class distribution and the ground-truth label.

Hard-label loss is the standard cross-entropy between a student's predictions at temperature T=1 and ground-truth labels. It anchors knowledge distillation in correctness while the soft-target term transfers the teacher's inter-class relationships. section 8.4.4

Use hard-label loss as one branch of the distillation objective:

  • Compute cross-entropy directly from the student's raw logits and labels; do not temperature-scale this term. section 8.6.2
  • Combine it with the temperature-scaled soft-target loss: L = αL_hard + (1−α)T²L_soft. section 8.4.4 section 8.6.2
  • Treat it as a safety net when teacher predictions contain errors or biases; a common weighting uses α=0.3. section 8.4.4

Numerical loss combination

For student logits [4.0, 2.5, 1.5, −0.5] and true class index 0, standard cross-entropy gives L_hard ≈ 0.275. With T=3, the soft loss is approximately 0.179; α=0.3 produces a combined loss of about 0.208. section 8.6.2

L = α L_hard + β T² KL(teacher_soft || student_soft)

Use the complete objective to see how hard-label cross-entropy supplies grounding alongside teacher-guided KL divergence. section 8.4.4

Listing 8.2 The knowledge distillation loss function.
import torch.nn.functional as F
def distillation_loss(student_logits,
     [CA] teacher_logits, labels,
     [CA] temperature=3.0, alpha=0.3):  #A
    soft_teacher = F.softmax(
     [CA] teacher_logits / temperature,
     [CA] dim=-1)  #B
    soft_student = F.log_softmax(
     [CA] student_logits / temperature,
     [CA] dim=-1)  #C
    kl_loss = F.kl_div(
     [CA] soft_student, soft_teacher,
     [CA] reduction='batchmean')  #D
 
    hard_loss = F.cross_entropy(
     [CA] student_logits, labels)  #E
    return (alpha * hard_loss
     [CA] + (1 - alpha)
     [CA] * temperature**2
     [CA] * kl_loss)  #F

The PyTorch listing places hard-label loss beside temperature-scaled KL divergence and returns their weighted sum. section 8.4.4

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains the definition, weighting, PyTorch implementation, gradient context, and numerical calculation within a combined distillation objective.
Examples
Calculates loss from four-class teacher and student logits, T=3 softmax outputs, α=0.3 weighting, and a PyTorch function.
What it assumes
Assumes student logits and ground-truth labels, while introducing their role alongside teacher logits, temperature, and KL divergence.
What you can do after
Apply it via PyTorch to compute the hard-label branch of a student-training objective.
How it connects
Connects cross-entropy with knowledge distillation, soft-target loss, KL divergence, temperature scaling, and ground-truth labels.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.