liveBook · Manning

Swish

As explained in GPU Programming with Triton

Swish is a smooth activation function defined as x multiplied by sigmoid(βx), with β controlling its shape.

Swish activation function
\[\text{Swish}_\beta(x) = x \cdot \sigma(\beta x) = \frac{x}{1 + e^{-\beta x}}\]

Swish multiplies the input by the sigmoid of β times that input. section 3.3.1

When β = 1, Swish is exactly SiLU. For other β values, the activation shapes differ, so the names are not interchangeable. section 3.3.1

In SwiGLU, Swish activates the gate projection, whose output is multiplied elementwise by the value projection. Fusing these operations avoids materializing intermediate results in global memory. section 3.3.1

Fused SwiGLU computation

Load gate values x1 and value values x2, compute x1 · sigmoid(x1), multiply by x2, and store the output. section 3.3.1

How GPU Programming with Triton covers it

How deep it goes
Defines Swish algebraically and distinguishes β = 1 from other values, without analyzing numerical behavior.
Examples
Uses gate and value projections in a fused SwiGLU kernel.
What you can do after
Understand when Swish and SiLU are interchangeable by checking whether β equals 1.
How it connects
Connects Swish to sigmoid, SiLU, gated linear units, SwiGLU, and matrix multiplication.

Also explained in

Compare how 4 books explain Swish →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.