A matrix multiplication kernel computes C = AB on a GPU. Matrix products underpin linear layers and attention projections, and can also express work in some convolution implementations. Performance depends not just on correct indexing, but on how data is accessed, reused, and mapped to GPU hardware. section 5.0 section 5.10
A practical implementation path moves from correctness to performance:
- Start with one program per output element. Accumulate its dot product over K in chunks, using explicit strides so inputs with different memory layouts can be addressed. section 5.5.2
- Move to tiled computation: each program computes an output tile and reuses loaded input values. Use tl.dot to contract K tiles into an FP32 accumulator. section 5.10
- Handle incomplete tiles with predicates; invalid K positions must contribute zero. Grouped program ordering and autotuning can help optimize cache reuse and tile configuration. section 5.10
Scalar baseline
For each output C[i, j], the naive kernel loads strips from row i of A and column j of B, multiplies them elementwise, reduces the products, and accumulates across K. It promotes FP16 or BF16 operands before multiplication so the products are accumulated in FP32. section 5.5.2
Listing 5.6 Naive matrix multiplication kernel, one output element per program
import torch
import triton
import triton.language as tl
@triton.jit
def naive_matmul_kernel(
a_ptr, b_ptr, c_ptr, #1
M, N, K, #2
stride_am, stride_ak, #3
stride_bk, stride_bn,
stride_cm, stride_cn,
BLOCK_K: tl.constexpr, #4
):
pid_m = tl.program_id(axis=0) #5
pid_n = tl.program_id(axis=1)
acc = 0.0 #6
for k_start in range(0, K, BLOCK_K): #7
offs_k = k_start + tl.arange(0, BLOCK_K) #8
a_ptrs = (
a_ptr + pid_m * stride_am
+ offs_k * stride_ak
) #9
b_ptrs = b_ptr + offs_k * stride_bk + pid_n * stride_bn
mask_k = offs_k < K #10
a = tl.load(a_ptrs, mask=mask_k, other=0.0)
b = tl.load(b_ptrs, mask=mask_k, other=0.0)
a_fp32 = a.to(tl.float32)
b_fp32 = b.to(tl.float32)
acc += tl.sum(a_fp32 * b_fp32, axis=0) #11
c_ptr_ij = c_ptr + pid_m * stride_cm + pid_n * stride_cn
tl.store(c_ptr_ij, acc) #12
Inspect the Triton baseline that assigns one program to each output element and advances through K in blocks. section 5.5.2
