SwiGLU multiplies a SiLU-activated gate projection by a second, value projection: SiLU(G) ⊙ U. With β = 1, SiLU is Swish and equals G · σ(G). section 3.3.1
After GEMM produces the gate and value tensors, SwiGLU is elementwise. Fusing its operations avoids materializing and rereading intermediate tensors in global memory. section 3.3.1
Listing 3..2 Fused SwiGLU kernel in Triton
@triton.jit
def swiglu(input_ptr, output_ptr, N, BLOCK_SIZE: tl.constexpr):
pid = tl.program_id(0) #1
H = N // 2 #2
offs = pid * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE) #3
mask = offs < H
x1 = tl.load(input_ptr + offs, mask=mask, other=0.0) #4
x2 = tl.load(input_ptr + H + offs, mask=mask, other=0.0) #5
silu = x1 * tl.sigmoid(x1) #6
out = silu * x2 #7
tl.store(output_ptr + offs, out, mask=mask) #8
The Triton kernel loads paired gate and value elements, computes SiLU and their product, then stores the result with masked accesses. section 3.3.1

