A soft target is a probability distribution over possible classes, rather than a one-hot hard label. In knowledge distillation, a frozen teacher produces this distribution for a training example, and a smaller student learns to match it. The probabilities communicate the teacher’s relative confidence across alternatives, not just its top prediction. section 8.2 section 8.2.1
A cat with alternatives
A hard label for a cat might be [1, 0, 0, 0]. A teacher might instead assign cat 86%, dog 12%, fox 1.5%, and bird 0.5%. The non-top probabilities expose the teacher’s learned relationships among classes: information often called dark knowledge. section 8.2.1
Temperature makes less likely alternatives easier to learn from. To compute temperature-scaled targets, divide the teacher’s logits by T, exponentiate them, and normalize. Raising T spreads probability mass across alternatives while preserving their ranking. At T=3, one example yields [0.523, 0.268, 0.138, 0.071] for cat, dog, fox, and bird; the larger alternative probabilities produce stronger learning signals. section 8.3.3 section 8.4.3
Figure 8.10 The complete step-by-step computation of soft targets at T=3: logits are divided by T, exponentiated, and normalized to produce the final probability distribution.

Follow the logits through division by temperature, exponentiation, and normalization to obtain the soft targets. section 8.3.3
Soft targets need not replace ground-truth labels. A combined distillation objective uses L = α L_hard + β L_soft T²: the hard-label term grounds training in the correct answers, while the soft-target term transfers the teacher’s distribution. In a comparison using the same student architecture, the student trained with both targets outperformed the hard-label baseline but did not fully match the teacher; the advantage was most pronounced on ambiguous examples. section 8.4.3 section 8.6.3
Classical distillation can generate soft targets by running teacher and student together, at the cost of holding both models in memory. An alternative is to generate and curate teacher responses offline, then fine-tune the student on that static dataset; this transfers generated text rather than the teacher’s probability distributions. section 8.5.3


