A soft target, or soft label, is a probability distribution over possible outputs rather than a single correct answer. In knowledge distillation, a teacher’s distribution lets a smaller student learn its confidence and alternative candidates, not just its preferred prediction. section 6.3 section 6.3.1
For next-token prediction, the teacher assigns probabilities across the vocabulary at each position. Temperature softens the distribution: higher values expose more alternatives when the teacher is very confident. A value near 2.0 is a practical starting point; lower values suit distributions that are already smooth. section 6.3.1 section 6.5.1
From logits to probabilities
For logits [2.0, 1.0, 0.1], softmax produces approximately [0.659, 0.242, 0.099]. The distribution retains the relative weight of all three candidates instead of identifying only the highest-scoring one. section 6.3.1
To train against soft targets, compare the student and teacher distributions with KL divergence. The PyTorch implementation applies log_softmax to student logits and softmax to teacher logits, both divided by temperature. It drops the last token position, flattens the remaining positions, uses batchmean reduction, and multiplies the loss by temperature squared. The student input is in log space to avoid numerical underflow. section 6.3.1
student_soft = F.log_softmax(
student_logits / temperature,
dim=-1) #A
teacher_soft = F.softmax(
teacher_logits / temperature,
dim=-1) #B
student_soft = student_soft[..., :-1, :].contiguous()
teacher_soft = teacher_soft[..., :-1, :].contiguous()
loss_logits = F.kl_div(
student_soft.view(-1, student_soft.size(-1)),
teacher_soft.view(-1, teacher_soft.size(-1)),
reduction='batchmean' #C
) * (temperature ** 2) #D
The listing implements temperature-scaled soft targets and KL-divergence loss for next-token predictions. section 6.3.1
Soft-target loss can be combined with cross-entropy on dataset hard labels and a hidden-state alignment loss. The weights control each signal’s influence: increase the hard-label weight for a high-quality, task-aligned dataset, or prioritize the soft-label weight when data is noisy or generic. A pure logits objective can also be effective; the balance depends on the task and experiment. section 6.3 section 6.5.1 section 6.6


