Hidden loss measures alignment between teacher and student intermediate representations during distillation. For each mapped layer pair, it averages one minus the mean cosine similarity of normalized activations, then averages across pairs. Its weight, gamma, controls its contribution to the compound loss. section 6.3.1
if gamma > 0:
loss_hidden = 0.0
num_aligned_layers = len(layer_map)
for student_idx, teacher_idx in enumerate(layer_map):
student_h = student_hiddens[student_idx]
teacher_h = teacher_hiddens[teacher_idx]
student_flat = student_h.reshape(
-1,
student_h.size(-1)) #A
teacher_flat = teacher_h.reshape(
-1,
teacher_h.size(-1)) #A
student_norm = F.normalize(
student_flat,
p=2,
dim=1) #B
teacher_norm = F.normalize(
teacher_flat,
p=2,
dim=1) #B
cos_sim = (student_norm * teacher_norm).sum(dim=1).mean() #C
loss_hidden += (1 - cos_sim) #D
loss_hidden = loss_hidden / num_aligned_layers #E
else:
loss_hidden = torch.tensor(0.0, device=student_logits.device)
See the PyTorch implementation for mapped-layer iteration, activation normalization, cosine similarity, and averaging. section 6.3.1
