A gate projection produces a control signal for the content projection in a gated feed-forward block. In a Llama-style MLP, gate_proj and up_proj process the same input in parallel. SiLU activates the gate signal, which is multiplied element-wise with the up_proj output; down_proj then contracts the filtered representation back to hidden_size. section 3.2.2
Figure 5.1 The complete GLU structure. Information enters with dimension hidden_size (2048 in Llama-3.2-1B) and expands to intermediate_size (8192) via gate_proj and up_proj. The gate_proj output, modulated by SiLU, is element-wise multiplied with up_proj’s output. The result is contracted back to hidden_size by down_proj.
The GLU diagram traces the parallel projections, element-wise multiplication, and contraction, including their dimensions in Llama-3.2-1B. section 5.1
Context-sensitive filtering
For the token “spring” in a sentence about mountain water, up_proj can represent several possible meanings. The gate uses the same context to modulate those components before contraction, favoring information relevant to the water-source meaning. section 3.2.2
The projections must stay aligned when reducing MLP width. Removing an intermediate neuron requires removing its counterpart from both gate_proj and up_proj, as well as the corresponding input to down_proj. This changes intermediate_size while preserving hidden_size at the block boundaries. section 5.1 section 5.2.2
Two selection approaches use the gate projection during width pruning:
- Static selection combines per-neuron weight scores from gate_proj and up_proj, then removes pairs with the lowest combined scores. section 5.1
- Data-driven selection combines normalized weight scores from all three projections with activation norms measured at the input to down_proj. Measuring before contraction preserves per-neuron traceability after gating. section 5.2
- For reconstruction, retain the highest-scoring indices, keep at least one neuron, and copy matching rows from gate_proj and up_proj and columns from down_proj into smaller layers. section 5.2.2
Calibration data shapes what the hybrid score favors. Limited domain coverage can leave other capabilities underrepresented, so pruning decisions should be interpreted in light of the calibration set. section 5.2 section 5.2.2
To inspect the gate directly, attach a forward hook to gate_proj. To capture the combined gated signal before compression, use a forward pre-hook on down_proj. Check module names for the model family: mismatched names can cause capture points to be skipped silently. section 10.1
