Task loss measures disagreement between a model’s predictions and hard labels for a specific task. In next-token prediction, cross-entropy trains the student directly on the correct tokens from the dataset. section 6.3.1 section 6.6
In knowledge distillation, task loss can anchor the student to the dataset while logit loss encourages it to imitate the teacher’s probability distribution. A compound objective can also include hidden loss to align intermediate representations. section 6.3.1 section 6.8
shift_logits = student_logits[..., :-1, :].contiguous() #A
shift_labels = labels[..., 1:].contiguous() #B
loss_task = F.cross_entropy(
shift_logits.view(-1, shift_logits.size(-1)), #C
shift_labels.view(-1),
ignore_index=-100 #D
)
The next-token implementation shifts student logits and labels before applying cross-entropy. It drops the last logit and first label, flattens the remaining positions, and ignores labels marked -100. section 6.3.1
Listing 6.6 General structure of the compound loss function
def compute_compound_loss(
student_logits,
teacher_logits,
student_hiddens,
teacher_hiddens,
labels,
layer_map,
alpha=0.4,
beta=0.4,
gamma=0.2,
temperature=2.0
):
loss_task = ...
loss_logits = ...
loss_hidden = ...
total_loss= alpha * loss_task + beta * loss_logits + gamma * loss_hidden
loss_dict = {
'total': total_loss.item(),
'task': loss_task.item(),
'logits': loss_logits.item(),
'hidden': loss_hidden.item()
}
return total_loss, loss_dict
The compound-loss structure weights task, logit, and hidden losses with alpha, beta, and gamma. It returns the total loss and separate scalar values for monitoring. section 6.3.1
A baseline for generative models assigns weights of 0.4 to task loss, 0.4 to logit loss, and 0.2 to hidden loss. Weights commonly sum to 1.0 for interpretability, although this is not mandatory; changing their combined scale also changes gradient magnitude. Adjust the balance to the objective rather than assuming hard labels are always necessary: reported Ministral experiments found that a pure distillation objective outperformed weighted combinations with next-token prediction loss. section 6.3.1 section 6.6
