liveBook · Manning

SwiGLU

As explained in GPU Programming with Triton

SwiGLU is a gated neural-network activation that multiplies a SiLU-activated linear projection by a second linear projection.

SwiGLU multiplies a SiLU-activated gate projection by a second, value projection: SiLU(G) ⊙ U. With β = 1, SiLU is Swish and equals G · σ(G). section 3.3.1

After GEMM produces the gate and value tensors, SwiGLU is elementwise. Fusing its operations avoids materializing and rereading intermediate tensors in global memory. section 3.3.1

Listing 3..2 Fused SwiGLU kernel in Triton
@triton.jit
def swiglu(input_ptr, output_ptr, N, BLOCK_SIZE: tl.constexpr):
    pid = tl.program_id(0) #1

    H = N // 2 #2
    offs = pid * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE) #3
    mask = offs < H

    x1 = tl.load(input_ptr + offs, mask=mask, other=0.0) #4
    x2 = tl.load(input_ptr + H + offs, mask=mask, other=0.0) #5

    silu = x1 * tl.sigmoid(x1) #6
    out = silu * x2 #7

    tl.store(output_ptr + offs, out, mask=mask) #8

The Triton kernel loads paired gate and value elements, computes SiLU and their product, then stores the result with masked accesses. section 3.3.1

How GPU Programming with Triton covers it

How deep it goes
Defines the activation, explains its memory-traffic motivation, and provides a fused Triton kernel.
Examples
Shows a Triton kernel that loads paired gate and value inputs and computes their activated product.
What it assumes
Assumes the gate and value projections are available as inputs from preceding matrix multiplication.
What you can do after
Build a fused elementwise SwiGLU kernel in Triton using masked loads and stores.
How it connects
Connects SwiGLU to the Gated Linear Unit, SiLU, Swish, GEMM, and Triton.

Also explained in

Compare how 2 books explain SwiGLU →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.