A multiplicative gate lets an MLP filter information according to its input. In a GLU, a second projection learns a per-neuron gate that modulates the content projection, replacing the fixed filtering of an activation such as GELU. section 3.2.2
Context-sensitive filtering
For “spring” in a water-related versus seasonal context, GELU applies the same activation; a learned gate can adapt its filtering to the input content. section 3.2.2
Figure 3.8 Processing flow in MLP Modules: classic vs. GLU. While the classic MLP follows a simple expansion-activation-contraction sequence, GLU uses two parallel projections: one up_proj that transforms the content and another gate_proj that creates a dynamic filter.
The diagram contrasts a classic expansion–activation–contraction MLP with GLU’s parallel content and gate projections. section 3.2.2
