A compound loss combines multiple loss components with configurable weights. For knowledge distillation, it lets a student learn from dataset labels, a teacher’s predictions, and corresponding intermediate representations at the same time. section 6.3.1 section 6.8
A typical objective is total_loss = alpha * loss_task + beta * loss_logits + gamma * loss_hidden. The weights control the emphasis on each signal; the example defaults are 0.4, 0.4, and 0.2. Keeping their sum near 1 makes experiments easier to compare and avoids changing the overall loss scale solely by changing the weights. section 6.3.1
Listing 6.6 General structure of the compound loss function
def compute_compound_loss(
student_logits,
teacher_logits,
student_hiddens,
teacher_hiddens,
labels,
layer_map,
alpha=0.4,
beta=0.4,
gamma=0.2,
temperature=2.0
):
loss_task = ...
loss_logits = ...
loss_hidden = ...
total_loss= alpha * loss_task + beta * loss_logits + gamma * loss_hidden
loss_dict = {
'total': total_loss.item(),
'task': loss_task.item(),
'logits': loss_logits.item(),
'hidden': loss_hidden.item()
}
return total_loss, loss_dict
This scaffold combines the three losses and returns both the training loss and individual values for monitoring. section 6.3.1
Compute the components before combining them:
- Task loss uses cross-entropy against hard dataset labels. Shift student logits and labels by one position so each prediction is compared with the next token; the example ignores labels marked -100. section 6.3.1
- Logits loss compares softened student and teacher distributions with KL divergence. Divide logits by temperature, use log-softmax for the student and softmax for the teacher, then multiply the loss by temperature squared. section 6.3.1
- Hidden loss compares mapped student and teacher layers. Flatten token representations, normalize them, compute one minus their mean cosine similarity, and average across aligned layers. Skip this component when gamma is zero. section 6.3.1
student_soft = F.log_softmax(
student_logits / temperature,
dim=-1) #A
teacher_soft = F.softmax(
teacher_logits / temperature,
dim=-1) #B
student_soft = student_soft[..., :-1, :].contiguous()
teacher_soft = teacher_soft[..., :-1, :].contiguous()
loss_logits = F.kl_div(
student_soft.view(-1, student_soft.size(-1)),
teacher_soft.view(-1, teacher_soft.size(-1)),
reduction='batchmean' #C
) * (temperature ** 2) #D
The logits-loss implementation shows the distribution transforms, sequence slicing, and temperature-squared multiplier. section 6.3.1
if gamma > 0:
loss_hidden = 0.0
num_aligned_layers = len(layer_map)
for student_idx, teacher_idx in enumerate(layer_map):
student_h = student_hiddens[student_idx]
teacher_h = teacher_hiddens[teacher_idx]
student_flat = student_h.reshape(
-1,
student_h.size(-1)) #A
teacher_flat = teacher_h.reshape(
-1,
teacher_h.size(-1)) #A
student_norm = F.normalize(
student_flat,
p=2,
dim=1) #B
teacher_norm = F.normalize(
teacher_flat,
p=2,
dim=1) #B
cos_sim = (student_norm * teacher_norm).sum(dim=1).mean() #C
loss_hidden += (1 - cos_sim) #D
loss_hidden = loss_hidden / num_aligned_layers #E
else:
loss_hidden = torch.tensor(0.0, device=student_logits.device)
The hidden-loss implementation shows how mapped layers are normalized and compared. section 6.3.1
Choose weights for the objective rather than assuming all three signals are always needed. Generative models can prioritize logits, while embedding models may need more hidden loss. Reported Mistral results favored pure logits distillation over weighted combinations; hard labels can still be useful when task-specific results are a priority. section 6.6 section 6.8
