Cosine distance measures directional difference between vectors: vectors [1, 2, 3] and [10, 20, 30] have the same direction despite different magnitudes. For hidden-state alignment, normalize student and teacher representation vectors, calculate one minus their cosine similarity, and average across mapped layers. section 6.3.1
if gamma > 0:
loss_hidden = 0.0
num_aligned_layers = len(layer_map)
for student_idx, teacher_idx in enumerate(layer_map):
student_h = student_hiddens[student_idx]
teacher_h = teacher_hiddens[teacher_idx]
student_flat = student_h.reshape(
-1,
student_h.size(-1)) #A
teacher_flat = teacher_h.reshape(
-1,
teacher_h.size(-1)) #A
student_norm = F.normalize(
student_flat,
p=2,
dim=1) #B
teacher_norm = F.normalize(
teacher_flat,
p=2,
dim=1) #B
cos_sim = (student_norm * teacher_norm).sum(dim=1).mean() #C
loss_hidden += (1 - cos_sim) #D
loss_hidden = loss_hidden / num_aligned_layers #E
else:
loss_hidden = torch.tensor(0.0, device=student_logits.device)
The implementation flattens batch and sequence dimensions, normalizes each representation, and averages cosine distance across aligned layers. section 6.3.1








