liveBook · Manning

Data-driven selection

As explained in Rearchitecting LLMs

Data-driven selection is the choice of model components based on their measured behavior on task-relevant data.

Data-driven selection chooses model blocks or neurons to keep by measuring their behavior on task-relevant data, rather than relying only on static weight-based criteria. section 4.3 section 5.2

For block selection, capture internal activations with PyTorch hooks and use cosine similarity to measure how much each block transforms information. For neuron selection in GLU layers, combine weight-based structural importance with activation norms from calibration data. The gate_proj and up_proj neurons are selected together, then smaller layers are reconstructed. section 4.3 section 5.2 section 5.5

Figure 5.5 Hybrid importance calculation in GLU architecture. The static component combines weight ranges from gate_proj, up_proj, and down_proj. The dynamic component captures activation norms at the input of down_proj, where gated information flows. Both components multiply to produce the final importance score.

Shows how structural scores and activation patterns combine to rank neurons. section 5.2

Calibration data changes what survives

Applying the same pruning percentage with WikiText or SMS calibration data produces models that retain neurons suited to formal text or conversational language, respectively. section 5.5

How Rearchitecting LLMs covers it

How deep it goes
Works from activation capture through block scoring and hybrid neuron ranking, with different scoring methods at each pruning level.
Examples
Compares WikiText- and SMS-calibrated models and uses Qwen3-0.6B for block analysis.
What it assumes
Assumes task-relevant calibration data; introduces PyTorch hooks from basic concepts, while dataset preparation is not repeated.
What you can do after
Apply PyTorch hooks and calibration data to rank model blocks or neurons for pruning.
How it connects
Connects model pruning with PyTorch hooks, cosine similarity, calibration data, and GLU architectures.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.