liveBook · Manning

Feature alignment

As explained in Rearchitecting LLMs

Feature alignment is a distillation technique that matches intermediate representations produced by corresponding blocks of teacher and student models.

Feature alignment transfers a teacher’s intermediate representations to a student by matching outputs from corresponding Transformer blocks. Unlike distillation based only on final labels or logits, it aims to make the student reach conclusions in similar ways. This is especially useful when intermediate representations are themselves the product, as in embedding models and RAG retrievers. section 6.2

When teacher and student have different numbers of blocks, choose a layer map:

  • Uniform mapping pairs blocks proportionally, giving the student access to early, middle, and late teacher layers. section 6.2
  • Last mapping pairs student blocks with the teacher’s final blocks; it is most relevant for aggressive depth pruning. section 6.2 section 6.6.1
  • Original mapping preserves the teacher-layer indices retained before pruning. section 6.2
Figure 6.2 The three mapping strategies: Uniform mapping distributes proportionally, last mapping aligns with the deepest layers, and original mapping preserves the original architectural connections, leaving gaps where the removed layers were.

Compare proportional, final-layer, and preserved-index mappings for a pruned student. section 6.2

For moderate pruning, uniform mapping can work well; the last-layer strategy is suggested when more than half the layers are removed. Alignment may add little when data is abundant and pruning is moderate, but can provide useful signal under aggressive compression or limited data. Setting gamma to zero disables the alignment loss and reduces computational cost; higher values are reserved for tasks that depend on intermediate representations. section 6.5.1 section 6.6.1

How Rearchitecting LLMs covers it

How deep it goes
Explains block-wise alignment, compares three layer maps, and discusses loss weighting and pruning conditions.
Examples
Compares 14-block students with 18-block teachers using uniform, last, and original layer maps.
What it assumes
Assumes teacher and student Transformer models whose layer counts may differ; explains how to map their blocks.
What you can do after
Apply feature alignment by choosing a student-to-teacher layer map and setting its loss weight for the target model.
How it connects
Connects feature alignment to knowledge distillation, hard-label and soft-label losses, layer pruning, and embedding models.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.