liveBook · Manning

Compound loss

As explained in Rearchitecting LLMs

A compound loss function combines multiple loss components, often using configurable weights.

A compound loss combines multiple loss components with configurable weights. For knowledge distillation, it lets a student learn from dataset labels, a teacher’s predictions, and corresponding intermediate representations at the same time. section 6.3.1 section 6.8

A typical objective is total_loss = alpha * loss_task + beta * loss_logits + gamma * loss_hidden. The weights control the emphasis on each signal; the example defaults are 0.4, 0.4, and 0.2. Keeping their sum near 1 makes experiments easier to compare and avoids changing the overall loss scale solely by changing the weights. section 6.3.1

Listing 6.6 General structure of the compound loss function
def compute_compound_loss(
    student_logits,
    teacher_logits,
    student_hiddens,
    teacher_hiddens,
    labels,
    layer_map,
    alpha=0.4,
    beta=0.4,
    gamma=0.2,
    temperature=2.0
):
 
    loss_task = ...
    loss_logits = ...
    loss_hidden = ...
 
    total_loss= alpha * loss_task + beta * loss_logits + gamma * loss_hidden
 
    loss_dict = {
         'total': total_loss.item(),
         'task': loss_task.item(),
         'logits': loss_logits.item(),
         'hidden': loss_hidden.item()
     }
 
     return total_loss, loss_dict
 

This scaffold combines the three losses and returns both the training loss and individual values for monitoring. section 6.3.1

Compute the components before combining them:

  • Task loss uses cross-entropy against hard dataset labels. Shift student logits and labels by one position so each prediction is compared with the next token; the example ignores labels marked -100. section 6.3.1
  • Logits loss compares softened student and teacher distributions with KL divergence. Divide logits by temperature, use log-softmax for the student and softmax for the teacher, then multiply the loss by temperature squared. section 6.3.1
  • Hidden loss compares mapped student and teacher layers. Flatten token representations, normalize them, compute one minus their mean cosine similarity, and average across aligned layers. Skip this component when gamma is zero. section 6.3.1
student_soft = F.log_softmax(
    student_logits / temperature, 
    dim=-1) #A
teacher_soft = F.softmax(
    teacher_logits / temperature, 
    dim=-1) #B
 
student_soft = student_soft[..., :-1, :].contiguous()
teacher_soft = teacher_soft[..., :-1, :].contiguous()
 
loss_logits = F.kl_div(
    student_soft.view(-1, student_soft.size(-1)),
    teacher_soft.view(-1, teacher_soft.size(-1)),
    reduction='batchmean' #C
) * (temperature ** 2) #D

The logits-loss implementation shows the distribution transforms, sequence slicing, and temperature-squared multiplier. section 6.3.1

if gamma > 0:
    loss_hidden = 0.0
    num_aligned_layers = len(layer_map)
 
    for student_idx, teacher_idx in enumerate(layer_map):
        student_h = student_hiddens[student_idx]
        teacher_h = teacher_hiddens[teacher_idx]
 
        student_flat = student_h.reshape(
           -1, 
           student_h.size(-1)) #A
        teacher_flat = teacher_h.reshape(
           -1, 
           teacher_h.size(-1)) #A
 
        student_norm = F.normalize(
           student_flat, 
           p=2, 
           dim=1) #B
        teacher_norm = F.normalize(
           teacher_flat, 
           p=2, 
           dim=1) #B
 
        cos_sim = (student_norm * teacher_norm).sum(dim=1).mean() #C
        loss_hidden += (1 - cos_sim) #D
 
    loss_hidden = loss_hidden / num_aligned_layers #E
else:
    loss_hidden = torch.tensor(0.0, device=student_logits.device)

The hidden-loss implementation shows how mapped layers are normalized and compared. section 6.3.1

Choose weights for the objective rather than assuming all three signals are always needed. Generative models can prioritize logits, while embedding models may need more hidden loss. Reported Mistral results favored pure logits distillation over weighted combinations; hard labels can still be useful when task-specific results are a priority. section 6.6 section 6.8

How Rearchitecting LLMs covers it

How deep it goes
Implements a weighted objective with token shifting, temperature-scaled KL divergence, and mapped hidden-state alignment.
Examples
Uses next-token prediction for “Work hard play hard” and a teacher distribution softened at temperature 2.0.
What you can do after
Apply it via PyTorch to combine and monitor task, logits, and hidden losses during distillation.
How it connects
Connects knowledge distillation with cross-entropy, KL divergence, temperature scaling, cosine similarity, and layer mapping.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.