liveBook · Manning

Up projection

As explained in Rearchitecting LLMs

An up projection is a feed-forward linear transformation that expands a hidden representation into an intermediate dimension.

An up projection (up_proj) is a linear layer that expands a hidden representation to an intermediate dimension inside a transformer MLP. In a gated linear unit (GLU), gate_proj expands the same input; its activated output multiplies the up projection output element by element. down_proj then contracts the result back to hidden_size. section 5.1

Llama-3.2-1B dimensions

Both up_proj and gate_proj map 2,048 hidden features to 8,192 intermediate features. SiLU activates the gate output before multiplication; down_proj maps the resulting 8,192 features back to 2,048. section 5.1 section 10.1

Figure 5.1 The complete GLU structure. Information enters with dimension hidden_size (2048 in Llama-3.2-1B) and expands to intermediate_size (8192) via gate_proj and up_proj. The gate_proj output, modulated by SiLU, is element-wise multiplied with up_proj’s output. The result is contracted back to hidden_size by down_proj.

Trace the two expansion paths, gating multiplication, and contraction through the GLU. section 5.1

Width pruning reduces the intermediate dimension while keeping hidden_size unchanged at module boundaries. The corresponding neurons in up_proj and gate_proj must be removed together to preserve gating alignment; the matching down_proj inputs must also be removed. section 5.1 section 5.2.2

A pruning workflow can select and rebuild the paired projections as follows:

  • Score neuron pairs using both projection weights. A static heuristic adds each pair's max + |min| weight scores and selects low-scoring pairs for removal. section 5.1
  • For data-driven selection, combine normalized weight scores from up_proj, gate_proj, and down_proj with activation norms measured at the input of down_proj. This captures gated intermediate signals before contraction mixes them. section 5.2
  • Keep the top k scores, sort their indices, and create smaller linear layers. Copy the selected rows of up_proj and gate_proj and the corresponding columns of down_proj; retain at least one intermediate neuron. section 5.2.2

Calibration data affects which neurons survive. Limited domain-specific data may miss neurons useful outside that domain, so activation-based selection requires care. For inspection, a forward hook can capture up_proj output; a pre-hook at down_proj captures the gated intermediate input instead. section 5.2 section 10.1

How Rearchitecting LLMs covers it

How deep it goes
Explains gated tensor flow, importance scoring, synchronized neuron selection, and reconstruction of smaller linear layers.
Examples
Illustrates expansion from 2,048 to 8,192 features in Llama-3.2-1B and calibration with WikiText or SMS.
What you can do after
Apply width pruning using PyTorch to select paired neurons and rebuild smaller MLP projections.
How it connects
Connects gated linear units, SiLU, neuron pruning, activation norms, and transformer attention.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.