A multilayer perceptron (MLP) is a feed-forward network that applies learned linear transformations and nonlinear activations. In a Transformer block, it follows attention and transforms contextualized representations using learned weights. The MLP can account for much of the model’s parameters and compute, making it a target for structural optimization. section 3.0 section 3.1.3
A classic Transformer MLP expands the hidden representation, applies a nonlinear activation, and projects it back to the original width. In DistilGPT2, the path is c_fc (768→3072), NewGELU, c_proj (3072→768), then dropout. Inspect forward(), not just the module listing: definition order need not match execution order. section 3.1.3
def forward(self, hidden_states: Optional[Tuple[torch.FloatTensor]]):
hidden_states = self.c_fc(hidden_states)
hidden_states = self.act(hidden_states)
hidden_states = self.c_proj(hidden_states)
hidden_states = self.dropout(hidden_states)
return hidden_states
Inspect the actual DistilGPT2 MLP forward sequence. section 3.1.3
A gated MLP uses two expansion projections. In Llama-3.2-1B, gate_proj and up_proj each map 2048→8192; SiLU is applied to the gate output, which is multiplied element-wise by the up output. down_proj then maps the result back to 2048. The nonlinearity enables patterns beyond simple linear relationships. section 5.1
Figure 5.1 The complete GLU structure. Information enters with dimension hidden_size (2048 in Llama-3.2-1B) and expands to intermediate_size (8192) via gate_proj and up_proj. The gate_proj output, modulated by SiLU, is element-wise multiplied with up_proj’s output. The result is contracted back to hidden_size by down_proj.
Trace the two expansion paths, gating multiplication, and contraction. section 5.1
Width pruning reduces the MLP’s intermediate dimension while preserving its input and output dimensions. For a gated MLP, select matching neuron indices in gate_proj and up_proj and retain the corresponding down_proj columns. One selection heuristic ranks pairs by their combined weight ranges; it is an assumption about importance, not a guarantee of preserved capability. section 5.1 section 5.1.2
Prune and check a Llama MLP
For each block, construct gate_proj and up_proj as 2048→k and down_proj as k→2048, copying the selected rows and columns. Replace the projections and update model.config.intermediate_size so downstream code sees the new shape. In one Llama-3.2-1B run, 40% intermediate-width pruning reduced total parameters by 26.06%; capability and performance still require evaluation. section 5.1.2 section 5.0
To inspect behavior rather than just structure, capture activations with PyTorch hooks. A forward pre-hook on down_proj captures the expanded representation before contraction; remove hooks after the run. MLPs can also be reused as mixture-of-experts components, although an expert transplanted from a model with different attention may need adjustment. section 10.1 section 9.6



