liveBook · Manning

Fused softmax kernel

As explained in GPU Programming with Triton

A fused softmax kernel computes the softmax reductions and elementwise operations in one GPU kernel while retaining intermediate values on chip.

A fused softmax computes the maximum, exponentials, sum, and normalization in one GPU kernel. Keeping intermediate values local avoids writing and rereading them in global memory and removes extra kernel launches. section 4.2.3

For a one-dimensional vector that fits in one program instance:

  • Mask out-of-range lanes with negative infinity so they do not affect the maximum; their exponentials become zero and do not affect the sum. section 4.2.3
  • Keep the maximum and denominator local, and retain the exponentials in registers until storing the normalized output. The num_warps setting controls thread-level parallelism, not correctness or tile shape. section 4.2.3
Listing 4.7 Fused softmax when the entire vector fits in one tile
import torch
import triton
import triton.language as tl

@triton.jit
def softmax_fused_1d_kernel(x_ptr, y_ptr, N, BLOCK_SIZE: tl.constexpr):
    offsets = tl.arange(0, BLOCK_SIZE)
    mask = offsets < N
    x = tl.load(x_ptr + offsets, mask=mask, other=-float("inf")).to(
        tl.float32)
    m = tl.max(x, axis=0)
    x = x - m
    num = tl.exp(x)
    den = tl.sum(num, axis=0)
    y = num / den
    tl.store(y_ptr + offsets, y, mask=mask)

def softmax_fused_1d(x: torch.Tensor, y: torch.Tensor, N: int):
    BLOCK_SIZE = triton.next_power_of_2(N) #1
    if BLOCK_SIZE > 1024:
        raise ValueError("This fused example assumes N <= 1024 so the full
         vector fits in one program instance.")

    num_warps = 8 if BLOCK_SIZE >= 512 else 4 #2
    softmax_fused_1d_kernel[(1,)](x, y, N, BLOCK_SIZE=BLOCK_SIZE,
    num_warps=num_warps)

See the Triton implementation, including power-of-two block sizing and the example’s limit of 1024 elements. section 4.2.3

How GPU Programming with Triton covers it

How deep it goes
Implements full-vector reductions, masking, register-held intermediates, and warp configuration for vectors fitting one program instance.
Examples
Demonstrates a Triton vector kernel with masked lanes and power-of-two block sizing, capped at 1024 elements.
What you can do after
Build a Triton kernel that computes one-dimensional softmax when the full vector fits in one program instance.
How it connects
Connects softmax with reductions, GPU kernels, Triton, registers, and global memory.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.