liveBook · Manning

Gate projection

As explained in Rearchitecting LLMs

A gate projection produces the signal that controls another intermediate signal in a gated feed-forward block.

A gate projection produces a control signal for the content projection in a gated feed-forward block. In a Llama-style MLP, gate_proj and up_proj process the same input in parallel. SiLU activates the gate signal, which is multiplied element-wise with the up_proj output; down_proj then contracts the filtered representation back to hidden_size. section 3.2.2

Figure 5.1 The complete GLU structure. Information enters with dimension hidden_size (2048 in Llama-3.2-1B) and expands to intermediate_size (8192) via gate_proj and up_proj. The gate_proj output, modulated by SiLU, is element-wise multiplied with up_proj’s output. The result is contracted back to hidden_size by down_proj.

The GLU diagram traces the parallel projections, element-wise multiplication, and contraction, including their dimensions in Llama-3.2-1B. section 5.1

Context-sensitive filtering

For the token “spring” in a sentence about mountain water, up_proj can represent several possible meanings. The gate uses the same context to modulate those components before contraction, favoring information relevant to the water-source meaning. section 3.2.2

The projections must stay aligned when reducing MLP width. Removing an intermediate neuron requires removing its counterpart from both gate_proj and up_proj, as well as the corresponding input to down_proj. This changes intermediate_size while preserving hidden_size at the block boundaries. section 5.1 section 5.2.2

Two selection approaches use the gate projection during width pruning:

  • Static selection combines per-neuron weight scores from gate_proj and up_proj, then removes pairs with the lowest combined scores. section 5.1
  • Data-driven selection combines normalized weight scores from all three projections with activation norms measured at the input to down_proj. Measuring before contraction preserves per-neuron traceability after gating. section 5.2
  • For reconstruction, retain the highest-scoring indices, keep at least one neuron, and copy matching rows from gate_proj and up_proj and columns from down_proj into smaller layers. section 5.2.2

Calibration data shapes what the hybrid score favors. Limited domain coverage can leave other capabilities underrepresented, so pruning decisions should be interpreted in light of the calibration set. section 5.2 section 5.2.2

To inspect the gate directly, attach a forward hook to gate_proj. To capture the combined gated signal before compression, use a forward pre-hook on down_proj. Check module names for the model family: mismatched names can cause capture points to be skipped silently. section 10.1

How Rearchitecting LLMs covers it

How deep it goes
Explains the gated computation and develops static and activation-based pruning through scoring, safety checks, and layer reconstruction.
Examples
Illustrates contextual filtering with “spring” and pruning with WikiText or SMS calibration data.
What you can do after
Apply width pruning with PyTorch by scoring paired neurons and reconstructing compatible MLP layers.
How it connects
Connects gate projection to GLUs, SiLU, element-wise multiplication, width pruning, and activation hooks.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.