liveBook · Manning

Down projection layer

As explained in Rearchitecting LLMs

A down projection is a linear transformation that reduces an intermediate representation back to the model's hidden dimension.

A down projection is the linear contraction at the end of a transformer MLP. It maps the expanded intermediate representation back to the model's hidden dimension, keeping the MLP output compatible with the next block. In a gated linear unit (GLU), it follows the element-wise multiplication of the content projection, up_proj, and the activated gate_proj. section 3.2.2 section 5.2.2

Llama-3.2-1B dimensions

The MLP expands a 2,048-dimensional hidden representation to 8,192 features through both gate_proj and up_proj. After gating, down_proj contracts the 8,192 features back to 2,048. A classic MLP has a corresponding contraction layer, named c_proj in DistilGPT2. section 3.2.2 section 5.1

(mlp): LlamaMLP(
    (gate_proj): Linear(in_features=2048, out_features=8192, bias=False)
    (up_proj): Linear(in_features=2048, out_features=8192, bias=False)
    (down_proj): Linear(in_features=8192, out_features=2048, bias=False)
    (act_fn): SiLU()
)

Inspect the three linear layers and their input and output dimensions. section 3.2.2

For width pruning, down_proj is both an observation point and part of the reconstruction. Its input retains the gated signal for each intermediate neuron; its output has already mixed and compressed those signals, making individual contributions harder to trace. section 5.2 section 5.2.1

A data-driven pruning workflow uses down_proj in three stages:

  • Capture activation norms at the input of each down_proj during calibration. A PyTorch hook can compute L2 norms and accumulate them on CPU rather than storing all activations on GPU. section 5.2.1
  • Score each intermediate neuron using both activation and structure. The structural component combines normalized weight-range scores from gate_proj, up_proj, and down_proj; multiplying it by the dynamic component favors neurons that are both structurally significant and active on the calibration data. section 5.2
  • Keep the highest-scoring neurons and rebuild the layers. Select corresponding rows in gate_proj and up_proj and columns in down_proj. The new down_proj input shrinks to the number kept, while its output remains hidden_size; a safety check prevents pruning every neuron. section 5.2.2
Listing 5.4 Configuring hooks to capture MLP activation norms
_accumulated_act_norms = {} #1
 
def setup_mlp_hooks_for_importance(model, device):
    global _accumulated_act_norms
    _accumulated_act_norms.clear() #2
    gc.collect() #2
    torch.cuda.empty_cache() #2
    handles = []
 
    for idx, layer in enumerate(model.model.layers):
        intermediate_size = layer.mlp.down_proj.in_features
        _accumulated_act_norms[idx] = torch.zeros(
            intermediate_size,
            dtype=torch.float32,
            device='cpu' #3
        )
 
    def make_hook(layer_idx): #4
        def hook(module, input, output): #5
 
            X_d = input[0].detach() #6
 
            act_norms_L2 = torch.norm(
                X_d.to(torch.float32),
                p=2,
                dim=(0, 1)  #7
            )
            _accumulated_act_norms[layer_idx] +=
 ➥act_norms_L2.cpu() #8
 
       return hook
 
    for idx, layer in enumerate(model.model.layers):
        handle = layer.mlp.down_proj.register_forward_hook(
            make_hook(idx)
        ) #9
        handles.append(handle)
 
    print(f"✓ Registered {len(handles)} hooks on down_proj")
 
    return handles
 
def get_activation_norms():
    return {
        layer_idx: norms.clone()
        for layer_idx, norms in _accumulated_act_norms.items()
    }

See how hooks calculate per-neuron norms at down_proj and accumulate them on CPU. section 5.2.1

Calibration data affects which neurons survive. A narrow domain can leave important neurons inactive during calibration, so activation alone can mistakenly remove capabilities outside that domain. The hybrid score also considers weight structure, but the retained neurons still depend on the calibration data. section 5.2 section 5.2.2

How Rearchitecting LLMs covers it

How deep it goes
Explains projection dimensions, activation hooks, hybrid neuron scoring, and reconstruction with coordinated weight indexing.
Examples
Uses Llama-3.2-1B's 8,192-to-2,048 contraction and contrasts WikiText and SMS calibration.
What you can do after
Apply it via PyTorch hooks and layer reconstruction to capture neuron activations and prune MLP width.
How it connects
Connects down projection to gated linear units, activation functions, neuron importance, calibration, and width pruning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.