Kernel fusion combines chained operations into fewer GPU kernels. Instead of writing each intermediate result to global memory and loading it again, a fused kernel keeps intermediates on chip, often in registers. It also removes launches between fused stages. The arithmetic remains mathematically equivalent, though floating-point results can differ slightly. section 3.3 section 4.2.3
SiLU followed by multiplication
For z[i] = x[i] × sigmoid(x[i]) × y[i], separate kernels first write the SiLU result and then read it back for multiplication. A fused kernel loads x[i] and y[i], computes the full expression in registers, and stores z[i]. For float32, the count falls from five to three global memory accesses per element: 20 to 12 bytes. If memory traffic determines runtime, the ratio suggests a theoretical 1.67× speedup, not a guaranteed result. section 3.3 section 3.7
The savings matter because registers are much faster than global memory. Reducing bytes moved while retaining the same arithmetic raises arithmetic intensity, giving the GPU less memory traffic to service for each computation. section 3.2.4 section 3.3
Fused softmax in Triton
A softmax pipeline can keep its maximum, exponentials, and denominator local rather than materializing them between kernels. The fused 1D example reads the input once, uses tl.max and tl.sum for the reductions, and writes the output once. It requires the entire axis to fit in one program instance; the supplied wrapper limits N to 1024. Masked loads use negative infinity, so padded lanes cannot affect the maximum and become zeros after exponentiation. The num_warps parameter adjusts thread-level parallelism without changing the tile’s logical shape. section 4.2.2 section 4.2.3
Listing 4.7 Fused softmax when the entire vector fits in one tile
import torch
import triton
import triton.language as tl
@triton.jit
def softmax_fused_1d_kernel(x_ptr, y_ptr, N, BLOCK_SIZE: tl.constexpr):
offsets = tl.arange(0, BLOCK_SIZE)
mask = offsets < N
x = tl.load(x_ptr + offsets, mask=mask, other=-float("inf")).to(
tl.float32)
m = tl.max(x, axis=0)
x = x - m
num = tl.exp(x)
den = tl.sum(num, axis=0)
y = num / den
tl.store(y_ptr + offsets, y, mask=mask)
def softmax_fused_1d(x: torch.Tensor, y: torch.Tensor, N: int):
BLOCK_SIZE = triton.next_power_of_2(N) #1
if BLOCK_SIZE > 1024:
raise ValueError("This fused example assumes N <= 1024 so the full
vector fits in one program instance.")
num_warps = 8 if BLOCK_SIZE >= 512 else 4 #2
softmax_fused_1d_kernel[(1,)](x, y, N, BLOCK_SIZE=BLOCK_SIZE,
num_warps=num_warps)
Inspect the fused softmax kernel and its single-instance launch. section 4.2.3
Fusion has a resource cost: keeping more values live can increase register use and reduce occupancy, the proportion of supported warps active on a GPU multiprocessor. Fewer active warps can make memory latency harder to hide. Higher occupancy alone does not guarantee better performance, so the saved traffic must be weighed against resource use. section 3.2.1



