A down projection is the linear contraction at the end of a transformer MLP. It maps the expanded intermediate representation back to the model's hidden dimension, keeping the MLP output compatible with the next block. In a gated linear unit (GLU), it follows the element-wise multiplication of the content projection, up_proj, and the activated gate_proj. section 3.2.2 section 5.2.2
Llama-3.2-1B dimensions
The MLP expands a 2,048-dimensional hidden representation to 8,192 features through both gate_proj and up_proj. After gating, down_proj contracts the 8,192 features back to 2,048. A classic MLP has a corresponding contraction layer, named c_proj in DistilGPT2. section 3.2.2 section 5.1
(mlp): LlamaMLP(
(gate_proj): Linear(in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(in_features=8192, out_features=2048, bias=False)
(act_fn): SiLU()
)
Inspect the three linear layers and their input and output dimensions. section 3.2.2
For width pruning, down_proj is both an observation point and part of the reconstruction. Its input retains the gated signal for each intermediate neuron; its output has already mixed and compressed those signals, making individual contributions harder to trace. section 5.2 section 5.2.1
A data-driven pruning workflow uses down_proj in three stages:
- Capture activation norms at the input of each down_proj during calibration. A PyTorch hook can compute L2 norms and accumulate them on CPU rather than storing all activations on GPU. section 5.2.1
- Score each intermediate neuron using both activation and structure. The structural component combines normalized weight-range scores from gate_proj, up_proj, and down_proj; multiplying it by the dynamic component favors neurons that are both structurally significant and active on the calibration data. section 5.2
- Keep the highest-scoring neurons and rebuild the layers. Select corresponding rows in gate_proj and up_proj and columns in down_proj. The new down_proj input shrinks to the number kept, while its output remains hidden_size; a safety check prevents pruning every neuron. section 5.2.2
Listing 5.4 Configuring hooks to capture MLP activation norms
_accumulated_act_norms = {} #1
def setup_mlp_hooks_for_importance(model, device):
global _accumulated_act_norms
_accumulated_act_norms.clear() #2
gc.collect() #2
torch.cuda.empty_cache() #2
handles = []
for idx, layer in enumerate(model.model.layers):
intermediate_size = layer.mlp.down_proj.in_features
_accumulated_act_norms[idx] = torch.zeros(
intermediate_size,
dtype=torch.float32,
device='cpu' #3
)
def make_hook(layer_idx): #4
def hook(module, input, output): #5
X_d = input[0].detach() #6
act_norms_L2 = torch.norm(
X_d.to(torch.float32),
p=2,
dim=(0, 1) #7
)
_accumulated_act_norms[layer_idx] +=
➥act_norms_L2.cpu() #8
return hook
for idx, layer in enumerate(model.model.layers):
handle = layer.mlp.down_proj.register_forward_hook(
make_hook(idx)
) #9
handles.append(handle)
print(f"✓ Registered {len(handles)} hooks on down_proj")
return handles
def get_activation_norms():
return {
layer_idx: norms.clone()
for layer_idx, norms in _accumulated_act_norms.items()
}
See how hooks calculate per-neuron norms at down_proj and accumulate them on CPU. section 5.2.1
Calibration data affects which neurons survive. A narrow domain can leave important neurons inactive during calibration, so activation alone can mistakenly remove capabilities outside that domain. The hybrid score also considers weight structure, but the retained neurons still depend on the calibration data. section 5.2 section 5.2.2
