Feature dynamics distillation (FDD) aligns how student and teacher representations change across mapped consecutive layers. Unlike layer-by-layer hidden-state alignment, it compares the difference between successive hidden states, focusing on the direction of change rather than absolute magnitude. section 6.4.2
for student_idx in range(len(layer_map) - 1): #A
teacher_idx = layer_map[student_idx]
teacher_idx_next = layer_map[student_idx + 1]
student_delta = student_hiddens[student_idx + 1] - #B
➥student_hiddens[student_idx]
teacher_delta = teacher_hiddens[teacher_idx_next] -
➥teacher_hiddens[teacher_idx]
student_delta_flat =
➥student_delta.reshape(-1, student_delta.size(-1)) #C
teacher_delta_flat = teacher_delta.reshape(-1, teacher_delta.size(-1))
student_delta_norm =
➥F.normalize(student_delta_flat, p=2, dim=1) #D
teacher_delta_norm = F.normalize(teacher_delta_flat, p=2, dim=1)
cos_sim_deriv = (student_delta_norm *
➥teacher_delta_norm).sum(dim=1).mean() #E
loss_derivative += (1 - cos_sim_deriv)
The implementation computes mapped student and teacher deltas, flattens and normalizes them, then accumulates one minus their mean cosine similarity. section 6.4.2
Advanced distillation at different data scales
In Gemma-3-270m experiments, advanced KD offered no advantage over labels-only at 2,000 samples. At 15,000 and 40,000 samples, it improved perplexity slightly, with capability retention remaining close to labels-only. These results concern advanced KD, not FDD in isolation. section 6.4
FDD can help use limited samples efficiently, particularly with high compression or specialized domains. It does not replace quality data or a sound pruning strategy. section 6.6.2
