An up projection (up_proj) is a linear layer that expands a hidden representation to an intermediate dimension inside a transformer MLP. In a gated linear unit (GLU), gate_proj expands the same input; its activated output multiplies the up projection output element by element. down_proj then contracts the result back to hidden_size. section 5.1
Llama-3.2-1B dimensions
Both up_proj and gate_proj map 2,048 hidden features to 8,192 intermediate features. SiLU activates the gate output before multiplication; down_proj maps the resulting 8,192 features back to 2,048. section 5.1 section 10.1
Figure 5.1 The complete GLU structure. Information enters with dimension hidden_size (2048 in Llama-3.2-1B) and expands to intermediate_size (8192) via gate_proj and up_proj. The gate_proj output, modulated by SiLU, is element-wise multiplied with up_proj’s output. The result is contracted back to hidden_size by down_proj.
Trace the two expansion paths, gating multiplication, and contraction through the GLU. section 5.1
Width pruning reduces the intermediate dimension while keeping hidden_size unchanged at module boundaries. The corresponding neurons in up_proj and gate_proj must be removed together to preserve gating alignment; the matching down_proj inputs must also be removed. section 5.1 section 5.2.2
A pruning workflow can select and rebuild the paired projections as follows:
- Score neuron pairs using both projection weights. A static heuristic adds each pair's max + |min| weight scores and selects low-scoring pairs for removal. section 5.1
- For data-driven selection, combine normalized weight scores from up_proj, gate_proj, and down_proj with activation norms measured at the input of down_proj. This captures gated intermediate signals before contraction mixes them. section 5.2
- Keep the top k scores, sort their indices, and create smaller linear layers. Copy the selected rows of up_proj and gate_proj and the corresponding columns of down_proj; retain at least one intermediate neuron. section 5.2.2
Calibration data affects which neurons survive. Limited domain-specific data may miss neurons useful outside that domain, so activation-based selection requires care. For inspection, a forward hook can capture up_proj output; a pre-hook at down_proj captures the gated intermediate input instead. section 5.2 section 10.1
