liveBook · Manning

Dark knowledge

As explained in Build a DeepSeek Model (From Scratch)

Dark knowledge is the informative structure in a teacher model's output probabilities beyond the most likely prediction, including relationships among alternative outputs.

Dark knowledge is the information in a teacher model’s output probabilities beyond its most likely prediction. The relative probabilities of alternative classes reveal relationships that a hard label discards. In knowledge distillation, these soft targets give a student a richer signal than the correct answer alone. section 8.2.1 section 8.9

A standard softmax can hide those relationships in tiny probabilities. Temperature scaling divides the logits by T before softmax: T > 1 spreads probability across alternatives while preserving their ranking. Too much smoothing makes the distribution nearly uniform and dilutes the useful signal. A typical starting range is T=2–5; smaller students may benefit from the lower end. section 8.3 section 8.3.5

Revealing alternatives

For cat, dog, fox, and bird logits [5, 3, 1, -1], softmax at T=1 gives approximately [0.865, 0.117, 0.016, 0.002]. At T=3 it gives [0.523, 0.268, 0.138, 0.071]: cat remains the top prediction, but the relative plausibility of dog, fox, and bird is clearer. At T=10, the distribution is nearly uniform. section 8.3.2

Figure 8.8 The same logits [5.0, 3.0, 1.0, -1.0] processed through softmax at three temperatures. At T=1, the distribution is too peaked, at T=3 the dark knowledge emerges clearly, and at T=10 the signal is lost in near-uniformity.

Compare the same logits at three temperatures to see the trade-off between a peaked and a nearly uniform distribution. section 8.3.2

To transfer dark knowledge, match teacher and student distributions at the same temperature using KL divergence. Combine that soft-target loss with cross-entropy against hard labels: L = α L_hard + β T² KL(teacher_soft || student_soft). The T² factor compensates for reduced soft-loss gradients at higher temperatures; the weights control reliance on ground truth versus the teacher. section 8.4.4 section 8.3.4

Listing 8.2 The knowledge distillation loss function.
import torch.nn.functional as F
def distillation_loss(student_logits,
     [CA] teacher_logits, labels,
     [CA] temperature=3.0, alpha=0.3):  #A
    soft_teacher = F.softmax(
     [CA] teacher_logits / temperature,
     [CA] dim=-1)  #B
    soft_student = F.log_softmax(
     [CA] student_logits / temperature,
     [CA] dim=-1)  #C
    kl_loss = F.kl_div(
     [CA] soft_student, soft_teacher,
     [CA] reduction='batchmean')  #D
 
    hard_loss = F.cross_entropy(
     [CA] student_logits, labels)  #E
    return (alpha * hard_loss
     [CA] + (1 - alpha)
     [CA] * temperature**2
     [CA] * kl_loss)  #F

The PyTorch loss computes teacher softmax, student log-softmax, KL divergence, and hard-label cross-entropy before combining them. section 8.4.4

Distillation need not always transfer dark knowledge through probabilities. An alternative for reasoning models is to fine-tune a student on teacher-generated reasoning traces rather than match temperature-scaled logits. section 8.5.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Works through temperature-scaled probabilities, the combined loss, gradient compensation, and student-capacity trade-offs.
Examples
Compares cat, dog, fox, and bird probabilities across temperatures and contrasts distilled and hard-label-only students.
What you can do after
Apply it via PyTorch by combining temperature-scaled KL divergence with hard-label cross-entropy.
How it connects
Connects knowledge distillation with softmax, temperature scaling, KL divergence, cross-entropy, and model capacity.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.