Element-wise fusion combines operations that act on corresponding tensor elements into one GPU computation. In SwiGLU, the gate and value projections are produced by preceding GEMMs; fusion combines the gate’s SiLU activation with multiplication by the value, avoiding separate intermediate tensor writes and reads. section 3.3.1
Simplified SwiGLU
The simplified formula shows the two element-wise operations fused over the gate and value tensors. section 3.3.1
Listing 3..2 Fused SwiGLU kernel in Triton
@triton.jit
def swiglu(input_ptr, output_ptr, N, BLOCK_SIZE: tl.constexpr):
pid = tl.program_id(0) #1
H = N // 2 #2
offs = pid * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE) #3
mask = offs < H
x1 = tl.load(input_ptr + offs, mask=mask, other=0.0) #4
x2 = tl.load(input_ptr + H + offs, mask=mask, other=0.0) #5
silu = x1 * tl.sigmoid(x1) #6
out = silu * x2 #7
tl.store(output_ptr + offs, out, mask=mask) #8
The Triton kernel loads both inputs, computes SiLU and multiplication, then stores the result with a mask. section 3.3.1
