A gated linear unit (GLU) gives a Transformer’s MLP an input-dependent filter. Instead of applying a fixed activation function to a single expansion, it uses a learned gate to modulate a parallel content projection, neuron by neuron. section 3.2.2
In a Llama-style GLU, gate_proj and up_proj independently expand the same input. SiLU activates the gate projection, which is multiplied element-wise by the content projection. The down_proj layer then contracts the result back to the model’s hidden size: down_proj(SiLU(gate_proj(x)) × up_proj(x)). section 5.1 section 10.2.2
Figure 5.1 The complete GLU structure. Information enters with dimension hidden_size (2048 in Llama-3.2-1B) and expands to intermediate_size (8192) via gate_proj and up_proj. The gate_proj output, modulated by SiLU, is element-wise multiplied with up_proj’s output. The result is contracted back to hidden_size by down_proj.
Follow the two parallel projections, their element-wise product, and the final contraction. section 5.1
Llama-3.2-1B MLP
Both gate_proj and up_proj expand a 2,048-dimensional input to 8,192 dimensions; down_proj maps the gated result back to 2,048. The SiLU-gated variant is called SwiGLU. Other variants include GeGLU, with GELU, and ReGLU, with ReLU. section 3.2.2 section 5.1
Width pruning targets the intermediate dimension while preserving the hidden size at the MLP boundary. Because gate_proj and up_proj operate as a pair, corresponding neurons must be removed together. One selection strategy adds each pair’s weight ranges; another combines structural importance with activation norms measured on calibration data. section 5.1 section 5.5
To inspect gating on real inputs, capture the activated gate, up_proj output, and down_proj input in one forward pass and compare their heatmaps. In one Llama example, the product attenuates the displayed signals; check the color scales before comparing panels, and do not assume the same pattern for every input or model. section 10.2.2 section 10.5
