liveBook · Manning

Gated linear unit

As explained in Rearchitecting LLMs

A gated linear unit is a neural-network structure that modulates one linear transformation with a learned gate derived from another transformation.

A gated linear unit (GLU) gives a Transformer’s MLP an input-dependent filter. Instead of applying a fixed activation function to a single expansion, it uses a learned gate to modulate a parallel content projection, neuron by neuron. section 3.2.2

In a Llama-style GLU, gate_proj and up_proj independently expand the same input. SiLU activates the gate projection, which is multiplied element-wise by the content projection. The down_proj layer then contracts the result back to the model’s hidden size: down_proj(SiLU(gate_proj(x)) × up_proj(x)). section 5.1 section 10.2.2

Figure 5.1 The complete GLU structure. Information enters with dimension hidden_size (2048 in Llama-3.2-1B) and expands to intermediate_size (8192) via gate_proj and up_proj. The gate_proj output, modulated by SiLU, is element-wise multiplied with up_proj’s output. The result is contracted back to hidden_size by down_proj.

Follow the two parallel projections, their element-wise product, and the final contraction. section 5.1

Llama-3.2-1B MLP

Both gate_proj and up_proj expand a 2,048-dimensional input to 8,192 dimensions; down_proj maps the gated result back to 2,048. The SiLU-gated variant is called SwiGLU. Other variants include GeGLU, with GELU, and ReGLU, with ReLU. section 3.2.2 section 5.1

Width pruning targets the intermediate dimension while preserving the hidden size at the MLP boundary. Because gate_proj and up_proj operate as a pair, corresponding neurons must be removed together. One selection strategy adds each pair’s weight ranges; another combines structural importance with activation norms measured on calibration data. section 5.1 section 5.5

To inspect gating on real inputs, capture the activated gate, up_proj output, and down_proj input in one forward pass and compare their heatmaps. In one Llama example, the product attenuates the displayed signals; check the color scales before comparing panels, and do not assume the same pattern for every input or model. section 10.2.2 section 10.5

How Rearchitecting LLMs covers it

How deep it goes
Explains projection shapes and gating, then develops synchronized pruning and activation visualization.
Examples
Uses Llama-3.2-1B projection shapes and a three-panel heatmap for the prompt “The cat sleeps.”
What you can do after
Diagnose GLU activations with heatmaps and apply synchronized width pruning using structural and calibration-based neuron scores.
How it connects
Connects GLUs to Transformer MLPs, activation functions, width pruning, and activation heatmaps.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.