liveBook · Manning

Hidden loss

As explained in Rearchitecting LLMs

A hidden loss is a loss term that measures disagreement between intermediate representations produced by a teacher and a student model.

Hidden loss measures alignment between teacher and student intermediate representations during distillation. For each mapped layer pair, it averages one minus the mean cosine similarity of normalized activations, then averages across pairs. Its weight, gamma, controls its contribution to the compound loss. section 6.3.1

if gamma > 0:
    loss_hidden = 0.0
    num_aligned_layers = len(layer_map)
 
    for student_idx, teacher_idx in enumerate(layer_map):
        student_h = student_hiddens[student_idx]
        teacher_h = teacher_hiddens[teacher_idx]
 
        student_flat = student_h.reshape(
           -1, 
           student_h.size(-1)) #A
        teacher_flat = teacher_h.reshape(
           -1, 
           teacher_h.size(-1)) #A
 
        student_norm = F.normalize(
           student_flat, 
           p=2, 
           dim=1) #B
        teacher_norm = F.normalize(
           teacher_flat, 
           p=2, 
           dim=1) #B
 
        cos_sim = (student_norm * teacher_norm).sum(dim=1).mean() #C
        loss_hidden += (1 - cos_sim) #D
 
    loss_hidden = loss_hidden / num_aligned_layers #E
else:
    loss_hidden = torch.tensor(0.0, device=student_logits.device)

See the PyTorch implementation for mapped-layer iteration, activation normalization, cosine similarity, and averaging. section 6.3.1

How Rearchitecting LLMs covers it

How deep it goes
Computes cosine-distance loss over mapped hidden-state layers and averages the results.
Examples
Aligns hidden activations from corresponding teacher and student Transformer blocks.
What it assumes
Assumes teacher and student hidden-state activations and a layer map pairing their blocks.
What you can do after
Apply via PyTorch to calculate a cosine-based alignment loss across mapped model layers.
How it connects
Connects knowledge distillation with layer mapping, task loss, and logits loss.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.