liveBook · Manning

Feature dynamics distillation

As explained in Rearchitecting LLMs

Feature dynamics distillation is a knowledge-distillation technique that aligns the changes in intermediate representations across consecutive layers of student and teacher models.

Feature dynamics distillation (FDD) aligns how student and teacher representations change across mapped consecutive layers. Unlike layer-by-layer hidden-state alignment, it compares the difference between successive hidden states, focusing on the direction of change rather than absolute magnitude. section 6.4.2

for student_idx in range(len(layer_map) - 1): #A
    teacher_idx = layer_map[student_idx]
    teacher_idx_next = layer_map[student_idx + 1]
 
    student_delta = student_hiddens[student_idx + 1] - #B
➥student_hiddens[student_idx]
    teacher_delta = teacher_hiddens[teacher_idx_next] - 
➥teacher_hiddens[teacher_idx]
 
    student_delta_flat = 
➥student_delta.reshape(-1, student_delta.size(-1)) #C
    teacher_delta_flat = teacher_delta.reshape(-1, teacher_delta.size(-1))
 
    student_delta_norm = 
➥F.normalize(student_delta_flat, p=2, dim=1) #D
    teacher_delta_norm = F.normalize(teacher_delta_flat, p=2, dim=1)
 
    cos_sim_deriv = (student_delta_norm * 
➥teacher_delta_norm).sum(dim=1).mean() #E
    loss_derivative += (1 - cos_sim_deriv)

The implementation computes mapped student and teacher deltas, flattens and normalizes them, then accumulates one minus their mean cosine similarity. section 6.4.2

Advanced distillation at different data scales

In Gemma-3-270m experiments, advanced KD offered no advantage over labels-only at 2,000 samples. At 15,000 and 40,000 samples, it improved perplexity slightly, with capability retention remaining close to labels-only. These results concern advanced KD, not FDD in isolation. section 6.4

FDD can help use limited samples efficiently, particularly with high compression or specialized domains. It does not replace quality data or a sound pruning strategy. section 6.6.2

How Rearchitecting LLMs covers it

How deep it goes
Computes mapped consecutive-layer deltas and aligns their directions with cosine similarity, while leaving data quality and pruning strategy essential.
Examples
Uses mapped student and teacher layer deltas and compares advanced KD with labels-only across three training-data sizes.
What it assumes
Assumes access to student and teacher hidden states and a mapping between their layers.
What you can do after
Apply FDD by calculating mapped consecutive-layer deltas and aligning their directions with cosine similarity.
How it connects
Connects knowledge distillation with hidden-state alignment, layer mapping, and Skew KL divergence.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.