An intermediate representation is a vector-valued activation produced inside a neural model between its input and output layers. In distillation, hidden states provide layer-level signals for aligning a student with a teacher. section 6.3.1
A layer map pairs student and teacher layers, which may have different indices. The hidden-state loss flattens and normalizes corresponding activations, then uses cosine similarity to measure their alignment. section 6.3.1
if gamma > 0:
loss_hidden = 0.0
num_aligned_layers = len(layer_map)
for student_idx, teacher_idx in enumerate(layer_map):
student_h = student_hiddens[student_idx]
teacher_h = teacher_hiddens[teacher_idx]
student_flat = student_h.reshape(
-1,
student_h.size(-1)) #A
teacher_flat = teacher_h.reshape(
-1,
teacher_h.size(-1)) #A
student_norm = F.normalize(
student_flat,
p=2,
dim=1) #B
teacher_norm = F.normalize(
teacher_flat,
p=2,
dim=1) #B
cos_sim = (student_norm * teacher_norm).sum(dim=1).mean() #C
loss_hidden += (1 - cos_sim) #D
loss_hidden = loss_hidden / num_aligned_layers #E
else:
loss_hidden = torch.tensor(0.0, device=student_logits.device)
The implementation accumulates cosine-distance losses across mapped student and teacher layers. section 6.3.1
Feature Dynamics Distillation compares changes between consecutive layers rather than only comparing absolute hidden states. It aligns the student and teacher deltas with cosine similarity, emphasizing the direction of change rather than its magnitude. section 6.4.2
for student_idx in range(len(layer_map) - 1): #A
teacher_idx = layer_map[student_idx]
teacher_idx_next = layer_map[student_idx + 1]
student_delta = student_hiddens[student_idx + 1] - #B
➥student_hiddens[student_idx]
teacher_delta = teacher_hiddens[teacher_idx_next] -
➥teacher_hiddens[teacher_idx]
student_delta_flat =
➥student_delta.reshape(-1, student_delta.size(-1)) #C
teacher_delta_flat = teacher_delta.reshape(-1, teacher_delta.size(-1))
student_delta_norm =
➥F.normalize(student_delta_flat, p=2, dim=1) #D
teacher_delta_norm = F.normalize(teacher_delta_flat, p=2, dim=1)
cos_sim_deriv = (student_delta_norm *
➥teacher_delta_norm).sum(dim=1).mean() #E
loss_derivative += (1 - cos_sim_deriv)
The code computes mapped layer-to-layer deltas, normalizes them, and accumulates their cosine-distance loss. section 6.4.2


