A fused softmax computes the maximum, exponentials, sum, and normalization in one GPU kernel. Keeping intermediate values local avoids writing and rereading them in global memory and removes extra kernel launches. section 4.2.3
For a one-dimensional vector that fits in one program instance:
- Mask out-of-range lanes with negative infinity so they do not affect the maximum; their exponentials become zero and do not affect the sum. section 4.2.3
- Keep the maximum and denominator local, and retain the exponentials in registers until storing the normalized output. The num_warps setting controls thread-level parallelism, not correctness or tile shape. section 4.2.3
Listing 4.7 Fused softmax when the entire vector fits in one tile
import torch
import triton
import triton.language as tl
@triton.jit
def softmax_fused_1d_kernel(x_ptr, y_ptr, N, BLOCK_SIZE: tl.constexpr):
offsets = tl.arange(0, BLOCK_SIZE)
mask = offsets < N
x = tl.load(x_ptr + offsets, mask=mask, other=-float("inf")).to(
tl.float32)
m = tl.max(x, axis=0)
x = x - m
num = tl.exp(x)
den = tl.sum(num, axis=0)
y = num / den
tl.store(y_ptr + offsets, y, mask=mask)
def softmax_fused_1d(x: torch.Tensor, y: torch.Tensor, N: int):
BLOCK_SIZE = triton.next_power_of_2(N) #1
if BLOCK_SIZE > 1024:
raise ValueError("This fused example assumes N <= 1024 so the full
vector fits in one program instance.")
num_warps = 8 if BLOCK_SIZE >= 512 else 4 #2
softmax_fused_1d_kernel[(1,)](x, y, N, BLOCK_SIZE=BLOCK_SIZE,
num_warps=num_warps)
See the Triton implementation, including power-of-two block sizing and the example’s limit of 1024 elements. section 4.2.3
