liveBook · Manning

Triton

As explained in GPU Programming with Triton

The Triton compiler transforms Triton programs into optimized GPU code for a target backend.

Triton is a Python-embedded domain-specific language and compiler for writing GPU kernels. It fills the gap between high-level frameworks such as PyTorch and lower-level programming with CUDA: you describe a blocked computation in Python, and the compiler generates optimized code for the target GPU. section 1.1.1 section 1.2.2

Triton is useful when a custom operation remains slow or unsupported after framework-level optimization, but the control and complexity of CUDA are unnecessary. Its kernels can approach handwritten CUDA performance, though performance depends on the workload. PyTorch’s TorchInductor can also generate Triton kernels without requiring you to write them directly. section 1.1.2 section 1.5

A kernel entry point is a Python function decorated with @triton.jit. On its first call, Triton compiles the function for the active GPU backend; the host then launches the compiled kernel over a grid of program instances. Each instance handles a block of work, while the compiler manages much of the thread-level execution, memory coalescing, shared memory use, and instruction scheduling. section 1.1.1 section 1.3.3 section 2.1

Figure 1.9 A Triton kernel’s path from source to execution. The host defines the kernel and compiles it into a resident binary on the first call, then launches that binary across the PCIe boundary; the device receives a grid of blocks and distributes them across its Streaming Multiprocessors as each one becomes free.
Triton kernel lifecycle from definition to execution

Follow a kernel from its first compilation on the host to grid execution on the GPU. section 1.4

Vector addition

The host passes two input vectors, their length, and a block size, then computes a launch grid. Each program uses its ID to construct offsets for one chunk, masks offsets beyond the vector length, loads valid input values, adds them, and stores the results with the same mask. Multiple program instances can run in parallel. section 2.2.9

The programmer still makes consequential choices, including tile shapes and the division of work across the grid. Hardware behavior remains relevant: for example, divergent branches can cause a warp to execute both paths sequentially. The compiler reduces low-level bookkeeping; it does not remove every performance concern. section 1.2.2 section 2.3

Listing 5.8 Tiled matmul kernel, assuming dimensions are exactly divisible by the tile sizes
import triton
import triton.language as tl

@triton.jit
def tiled_matmul_kernel(
    a_ptr, b_ptr, c_ptr,
    M, N, K,
    stride_am, stride_ak,
    stride_bk, stride_bn,
    stride_cm, stride_cn,
    BLOCK_M: tl.constexpr, #1
    BLOCK_N: tl.constexpr,
    BLOCK_K: tl.constexpr,
):
    pid_m = tl.program_id(axis=0) #2
    pid_n = tl.program_id(axis=1)

    offs_m = (
        pid_m * BLOCK_M + tl.arange(0, BLOCK_M)
    ) #3
    offs_n = pid_n * BLOCK_N + tl.arange(0, BLOCK_N)
    offs_k = tl.arange(0, BLOCK_K)

    a_ptrs = (
        a_ptr
        + offs_m[:, None] * stride_am
        + offs_k[None, :] * stride_ak
    ) #4
    b_ptrs = (
        b_ptr
        + offs_k[:, None] * stride_bk
        + offs_n[None, :] * stride_bn
    ) #4

    acc = tl.zeros(
        (BLOCK_M, BLOCK_N), dtype=tl.float32
    ) #5

    for k_start in range(0, K, BLOCK_K): #6
        a = tl.load(a_ptrs)
        b = tl.load(b_ptrs)
        acc = tl.dot(a, b, acc) #7
        a_ptrs += BLOCK_K * stride_ak #8
        b_ptrs += BLOCK_K * stride_bk

    c_ptrs = (
        c_ptr
        + offs_m[:, None] * stride_cm
        + offs_n[None, :] * stride_cn
    ) #9
    tl.store(c_ptrs, acc)

Inspect a tiled matrix multiplication kernel that uses program IDs, pointer tiles, an FP32 accumulator, and tl.dot; its initial form requires dimensions divisible by the tile sizes. section 5.5.5

The Python package alone is not enough: the matching GPU runtime must be installed. The NVIDIA backend requires CUDA 11.8 or later, while the AMD backend uses ROCm. Triton the GPU programming language is unrelated to NVIDIA Triton Inference Server. section 1.1.1 section 1.2.3

How GPU Programming with Triton covers it

How deep it goes
Explains the programming model, JIT execution, and complete kernels, including masked vector addition and tiled matrix multiplication.
Examples
Uses vector addition and tiled matrix multiplication to demonstrate program IDs, offsets, masks, tile loads, and accumulation.
What it assumes
Assumes intermediate Python but introduces GPU programming without requiring advanced CUDA experience.
What you can do after
Builds basic GPU kernels with Triton and launches, debugs, and benchmarks them from Python.
How it connects
Connects GPU kernels to PyTorch, TorchInductor, CUDA, GPU execution grids, and shared memory.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.