liveBook · Manning

Element-wise fusion

As explained in GPU Programming with Triton

Element-wise fusion combines multiple element-wise tensor operations into one GPU computation.

Element-wise fusion combines operations that act on corresponding tensor elements into one GPU computation. In SwiGLU, the gate and value projections are produced by preceding GEMMs; fusion combines the gate’s SiLU activation with multiplication by the value, avoiding separate intermediate tensor writes and reads. section 3.3.1

Simplified SwiGLU
\[\mathrm{SwiGLU}(x) = \underbrace{\mathrm{SiLU}(G)}_{\text{Gate}} \odot \underbrace{U}_{\text{Value}}\]

The simplified formula shows the two element-wise operations fused over the gate and value tensors. section 3.3.1

Listing 3..2 Fused SwiGLU kernel in Triton
@triton.jit
def swiglu(input_ptr, output_ptr, N, BLOCK_SIZE: tl.constexpr):
    pid = tl.program_id(0) #1

    H = N // 2 #2
    offs = pid * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE) #3
    mask = offs < H

    x1 = tl.load(input_ptr + offs, mask=mask, other=0.0) #4
    x2 = tl.load(input_ptr + H + offs, mask=mask, other=0.0) #5

    silu = x1 * tl.sigmoid(x1) #6
    out = silu * x2 #7

    tl.store(output_ptr + offs, out, mask=mask) #8

The Triton kernel loads both inputs, computes SiLU and multiplication, then stores the result with a mask. section 3.3.1

How GPU Programming with Triton covers it

How deep it goes
Traces SwiGLU from its simplified formula through intermediate memory costs to a masked Triton kernel.
Examples
Uses SwiGLU’s SiLU-gated multiplication and a Triton kernel that loads, computes, and stores masked elements.
What you can do after
Build it from scratch by fusing SiLU and multiplication over gate and value tensors in a Triton kernel.
How it connects
Connects element-wise fusion to SwiGLU, SiLU, sigmoid, GEMM, and Triton.

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.