liveBook · Manning

Multiplicative gate

As explained in Rearchitecting LLMs

A multiplicative gate is a learned vector that scales another representation element by element to control information flow.

A multiplicative gate lets an MLP filter information according to its input. In a GLU, a second projection learns a per-neuron gate that modulates the content projection, replacing the fixed filtering of an activation such as GELU. section 3.2.2

Context-sensitive filtering

For “spring” in a water-related versus seasonal context, GELU applies the same activation; a learned gate can adapt its filtering to the input content. section 3.2.2

Figure 3.8 Processing flow in MLP Modules: classic vs. GLU. While the classic MLP follows a simple expansion-activation-contraction sequence, GLU uses two parallel projections: one up_proj that transforms the content and another gate_proj that creates a dynamic filter.

The diagram contrasts a classic expansion–activation–contraction MLP with GLU’s parallel content and gate projections. section 3.2.2

How Rearchitecting LLMs covers it

How deep it goes
Explains per-neuron gating and parallel projections, then contrasts GLU with fixed-activation MLP filtering.
Examples
Contrasts “spring” in water and seasonal contexts and compares DistilGPT2 and Llama-3.2-1B MLP structures.
What you can do after
Understand how an input-dependent gate adapts MLP filtering compared with a fixed activation.
How it connects
Connects multiplicative gates to MLPs, GLU, GELU, SiLU, SwiGLU, GeGLU, and ReGLU.

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.