liveBook · Manning

Multilayer perceptron

As explained in Rearchitecting LLMs

A multilayer perceptron is a feed-forward neural network composed of successive learned linear transformations and nonlinear activations.

A multilayer perceptron (MLP) is a feed-forward network that applies learned linear transformations and nonlinear activations. In a Transformer block, it follows attention and transforms contextualized representations using learned weights. The MLP can account for much of the model’s parameters and compute, making it a target for structural optimization. section 3.0 section 3.1.3

A classic Transformer MLP expands the hidden representation, applies a nonlinear activation, and projects it back to the original width. In DistilGPT2, the path is c_fc (768→3072), NewGELU, c_proj (3072→768), then dropout. Inspect forward(), not just the module listing: definition order need not match execution order. section 3.1.3

def forward(self, hidden_states: Optional[Tuple[torch.FloatTensor]]):
        hidden_states = self.c_fc(hidden_states)
        hidden_states = self.act(hidden_states)
        hidden_states = self.c_proj(hidden_states)
        hidden_states = self.dropout(hidden_states)
        return hidden_states

Inspect the actual DistilGPT2 MLP forward sequence. section 3.1.3

A gated MLP uses two expansion projections. In Llama-3.2-1B, gate_proj and up_proj each map 2048→8192; SiLU is applied to the gate output, which is multiplied element-wise by the up output. down_proj then maps the result back to 2048. The nonlinearity enables patterns beyond simple linear relationships. section 5.1

Figure 5.1 The complete GLU structure. Information enters with dimension hidden_size (2048 in Llama-3.2-1B) and expands to intermediate_size (8192) via gate_proj and up_proj. The gate_proj output, modulated by SiLU, is element-wise multiplied with up_proj’s output. The result is contracted back to hidden_size by down_proj.

Trace the two expansion paths, gating multiplication, and contraction. section 5.1

Width pruning reduces the MLP’s intermediate dimension while preserving its input and output dimensions. For a gated MLP, select matching neuron indices in gate_proj and up_proj and retain the corresponding down_proj columns. One selection heuristic ranks pairs by their combined weight ranges; it is an assumption about importance, not a guarantee of preserved capability. section 5.1 section 5.1.2

Prune and check a Llama MLP

For each block, construct gate_proj and up_proj as 2048→k and down_proj as k→2048, copying the selected rows and columns. Replace the projections and update model.config.intermediate_size so downstream code sees the new shape. In one Llama-3.2-1B run, 40% intermediate-width pruning reduced total parameters by 26.06%; capability and performance still require evaluation. section 5.1.2 section 5.0

To inspect behavior rather than just structure, capture activations with PyTorch hooks. A forward pre-hook on down_proj captures the expanded representation before contraction; remove hooks after the run. MLPs can also be reused as mixture-of-experts components, although an expert transplanted from a model with different attention may need adjustment. section 10.1 section 9.6

How Rearchitecting LLMs covers it

How deep it goes
Explains classic and gated forward paths, tensor dimensions, paired pruning, reconstruction, and hook placement.
Examples
Uses DistilGPT2’s 768→3072→768 MLP and Llama-3.2-1B’s gated 2048→8192→2048 MLP.
What you can do after
Operate/configure gated MLP width pruning and inspect intermediate activations with PyTorch hooks.
How it connects
Connects MLPs to attention, gated linear units, width pruning, activations, and mixture of experts.

Also explained in

Compare how 4 books explain multilayer perceptron →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.