Data-driven selection chooses model blocks or neurons to keep by measuring their behavior on task-relevant data, rather than relying only on static weight-based criteria. section 4.3 section 5.2
For block selection, capture internal activations with PyTorch hooks and use cosine similarity to measure how much each block transforms information. For neuron selection in GLU layers, combine weight-based structural importance with activation norms from calibration data. The gate_proj and up_proj neurons are selected together, then smaller layers are reconstructed. section 4.3 section 5.2 section 5.5
Figure 5.5 Hybrid importance calculation in GLU architecture. The static component combines weight ranges from gate_proj, up_proj, and down_proj. The dynamic component captures activation norms at the input of down_proj, where gated information flows. Both components multiply to produce the final importance score.
Shows how structural scores and activation patterns combine to rank neurons. section 5.2
Calibration data changes what survives
Applying the same pruning percentage with WikiText or SMS calibration data produces models that retain neurons suited to formal text or conversational language, respectively. section 5.5
