liveBook · Manning

Swish

As explained in Rearchitecting LLMs

SiLU is a smooth activation function defined by multiplying an input by its sigmoid.

Swish, also called SiLU, is a smooth activation that multiplies an input by its sigmoid. In a GLU MLP, it modulates the gate_proj output before that output is multiplied element-wise with up_proj. section 5.1

Figure 5.1 The complete GLU structure. Information enters with dimension hidden_size (2048 in Llama-3.2-1B) and expands to intermediate_size (8192) via gate_proj and up_proj. The gate_proj output, modulated by SiLU, is element-wise multiplied with up_proj’s output. The result is contracted back to hidden_size by down_proj.

Trace how a GLU MLP expands hidden_size through gate_proj and up_proj, applies SiLU to the gate, multiplies the outputs, then contracts through down_proj. section 5.1

How Rearchitecting LLMs covers it

How deep it goes
Defines the input-times-sigmoid activation and explains its role in GLU gating, without detailing its implementation.
Examples
Uses Llama-3.2-1B's 2048-to-8192 GLU expansion to show SiLU-gated outputs multiplied before down_proj.
What it assumes
Assumes familiarity with MLP projections while briefly reviewing GLU gating and paired-neuron coordination.
What you can do after
Understand how SiLU gates one GLU projection before element-wise multiplication with its paired projection.
How it connects
Connects Swish to GLU architectures, MLP layers, and coordinated gate_proj and up_proj projections.

Also explained in

Compare how 4 books explain Swish →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.