Feature alignment transfers a teacher’s intermediate representations to a student by matching outputs from corresponding Transformer blocks. Unlike distillation based only on final labels or logits, it aims to make the student reach conclusions in similar ways. This is especially useful when intermediate representations are themselves the product, as in embedding models and RAG retrievers. section 6.2
When teacher and student have different numbers of blocks, choose a layer map:
- Uniform mapping pairs blocks proportionally, giving the student access to early, middle, and late teacher layers. section 6.2
- Last mapping pairs student blocks with the teacher’s final blocks; it is most relevant for aggressive depth pruning. section 6.2 section 6.6.1
- Original mapping preserves the teacher-layer indices retained before pruning. section 6.2
Figure 6.2 The three mapping strategies: Uniform mapping distributes proportionally, last mapping aligns with the deepest layers, and original mapping preserves the original architectural connections, leaving gaps where the removed layers were.
Compare proportional, final-layer, and preserved-index mappings for a pruned student. section 6.2
For moderate pruning, uniform mapping can work well; the last-layer strategy is suggested when more than half the layers are removed. Alignment may add little when data is abundant and pruning is moderate, but can provide useful signal under aggressive compression or limited data. Setting gamma to zero disables the alignment loss and reduces computational cost; higher values are reserved for tasks that depend on intermediate representations. section 6.5.1 section 6.6.1
