liveBook · Manning

SwiGLU

SwiGLU is a gated linear-unit activation variant that multiplies a SiLU-activated linear projection by a second learned linear projection.

How 2 Manning books explain it

GPU Programming with Triton

SwiGLU is a gated neural-network activation that multiplies a SiLU-activated linear projection by a second linear projection.

How it is used
Uses SwiGLU to demonstrate kernel fusion for an elementwise activation following matrix multiplication.
How deep it goes
Defines the activation, explains its memory-traffic motivation, and provides a fused Triton kernel.
Examples
Shows a Triton kernel that loads paired gate and value inputs and computes their activated product.
What you can do after
Build a fused elementwise SwiGLU kernel in Triton using masked loads and stores.

Read how GPU Programming with Triton explains SwiGLU →

LLMs in Production

SWIGLU is a gated activation function that uses the Swish function to modulate one linear projection with another.

How it is used
Uses SwiGLU as the nonlinear activation throughout a simplified Llama 3-style model.
How deep it goes
Explains its activation role and placement, but defers the internal implementation to a notebook.
Examples
Demonstrates SwiGLU inside LlamaBlock and the final SimpleLlama feed-forward network.
What you can do after
Apply it in a Llama-style PyTorch feed-forward network in place of ReLU.

Read how LLMs in Production explains SwiGLU →

Related concepts

Questions about SwiGLU

What is SwiGLU?

SwiGLU is a gated linear-unit activation variant that multiplies a SiLU-activated linear projection by a second learned linear projection.

Which Manning books explain SwiGLU?

GPU Programming with Triton and LLMs in Production.

What is the difference between SwiGLU and ReLU?

SwiGLU is a gated linear-unit activation variant that multiplies a SiLU-activated linear projection by a second learned linear projection. ReLU is an activation function that maps an input x to the greater of x and zero.

What is SwiGLU a kind of?

SwiGLU is a kind of gated linear unit. A gated linear unit is a neural-network structure that combines a linear transformation with a gate derived from another transformation through element-wise interaction.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.