liveBook · Manning

Soft target

As explained in Rearchitecting LLMs

A soft label is a probability-weighted target distribution that represents graded model predictions rather than a single discrete target.

A soft target, or soft label, is a probability distribution over possible outputs rather than a single correct answer. In knowledge distillation, a teacher’s distribution lets a smaller student learn its confidence and alternative candidates, not just its preferred prediction. section 6.3 section 6.3.1

For next-token prediction, the teacher assigns probabilities across the vocabulary at each position. Temperature softens the distribution: higher values expose more alternatives when the teacher is very confident. A value near 2.0 is a practical starting point; lower values suit distributions that are already smooth. section 6.3.1 section 6.5.1

From logits to probabilities

For logits [2.0, 1.0, 0.1], softmax produces approximately [0.659, 0.242, 0.099]. The distribution retains the relative weight of all three candidates instead of identifying only the highest-scoring one. section 6.3.1

To train against soft targets, compare the student and teacher distributions with KL divergence. The PyTorch implementation applies log_softmax to student logits and softmax to teacher logits, both divided by temperature. It drops the last token position, flattens the remaining positions, uses batchmean reduction, and multiplies the loss by temperature squared. The student input is in log space to avoid numerical underflow. section 6.3.1

student_soft = F.log_softmax(
    student_logits / temperature, 
    dim=-1) #A
teacher_soft = F.softmax(
    teacher_logits / temperature, 
    dim=-1) #B
 
student_soft = student_soft[..., :-1, :].contiguous()
teacher_soft = teacher_soft[..., :-1, :].contiguous()
 
loss_logits = F.kl_div(
    student_soft.view(-1, student_soft.size(-1)),
    teacher_soft.view(-1, teacher_soft.size(-1)),
    reduction='batchmean' #C
) * (temperature ** 2) #D

The listing implements temperature-scaled soft targets and KL-divergence loss for next-token predictions. section 6.3.1

Soft-target loss can be combined with cross-entropy on dataset hard labels and a hidden-state alignment loss. The weights control each signal’s influence: increase the hard-label weight for a high-quality, task-aligned dataset, or prioritize the soft-label weight when data is noisy or generic. A pure logits objective can also be effective; the balance depends on the task and experiment. section 6.3 section 6.5.1 section 6.6

How Rearchitecting LLMs covers it

How deep it goes
Implements temperature-scaled KL divergence and gives configuration guidance for balancing it against hard-label and hidden-state losses.
Examples
Uses a three-logit softmax calculation and next-token vocabulary distributions to illustrate graded predictions.
What you can do after
Apply soft-target distillation via PyTorch’s softmax, log_softmax, and kl_div functions within a composite loss.
How it connects
Connects knowledge distillation with logits, temperature, KL divergence, cross-entropy, hard labels, and hidden-state alignment.

Also explained in

Compare how 3 books explain soft target →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.