tl.max finds the maximum along a dimension of a Triton tensor tile. Its axis applies to that tile, which belongs to one program instance; it does not combine values across program instances. A grid-wide maximum requires coordinating partial results, such as with another kernel or atomic operations. section 4.1.2
In vector softmax, a program instance can load the full vector and use tl.max to find the value subtracted before exponentiation. Keeping the reduction local lets the pipeline stay in one kernel, at the cost of using more on-chip resources for larger tiles. section 4.2.2
Listing 4.1 Row-wise sum and max using block accumulation
import torch
import triton
import triton.language as tl
@triton.jit
def row_sum_max_kernel(x_ptr, out_sum_ptr, out_max_ptr,
stride_x,
N_COLS: tl.constexpr,
BLOCK_SIZE: tl.constexpr):
row = tl.program_id(0)
x_row_ptr = x_ptr + row * stride_x
acc_sum = tl.zeros([BLOCK_SIZE], dtype=tl.float32) #1
acc_max = tl.full([BLOCK_SIZE], float("-inf"), dtype=tl.float32) #2
neg_inf = tl.full([BLOCK_SIZE], float("-inf"), dtype=tl.float32)
for off in tl.static_range(0, N_COLS, BLOCK_SIZE): #3
cols = off + tl.arange(0, BLOCK_SIZE)
mask = cols < N_COLS
x = tl.load(x_row_ptr + cols, mask=mask, other=0.0).to(tl.float32)
acc_sum += x
acc_max = tl.maximum(acc_max, tl.where(mask, x, neg_inf)) #4
row_sum = tl.sum(acc_sum, axis=0) #5
row_max = tl.max(acc_max, axis=0)
tl.store(out_sum_ptr + row, row_sum)
tl.store(out_max_ptr + row, row_max)
def row_sum_max(x: torch.Tensor, block_size: int = 1024):
assert x.is_cuda and x.ndim == 2
n_rows, n_cols = x.shape
out_sum = torch.empty((n_rows,), device=x.device, dtype=torch.float32)
out_max = torch.empty((n_rows,), device=x.device, dtype=torch.float32)
grid = (n_rows,)
row_sum_max_kernel[grid](
x, out_sum, out_max,
x.stride(0),
N_COLS=n_cols,
BLOCK_SIZE=block_size,
num_warps=4,
)
return out_sum, out_max
See a row-wise kernel that accumulates a maximum across blocks of columns and handles out-of-range lanes with a mask. section 4.1.2
