Swish, also called SiLU, is a smooth activation that multiplies an input by its sigmoid. In a GLU MLP, it modulates the gate_proj output before that output is multiplied element-wise with up_proj. section 5.1
Figure 5.1 The complete GLU structure. Information enters with dimension hidden_size (2048 in Llama-3.2-1B) and expands to intermediate_size (8192) via gate_proj and up_proj. The gate_proj output, modulated by SiLU, is element-wise multiplied with up_proj’s output. The result is contracted back to hidden_size by down_proj.
Trace how a GLU MLP expands hidden_size through gate_proj and up_proj, applies SiLU to the gate, multiplies the outputs, then contracts through down_proj. section 5.1



