liveBook · Manning

Program instance

As explained in GPU Programming with Triton

A program instance is Triton's unit of work corresponding to one block of input elements in a compiled kernel launch.

A program instance is Triton’s unit of work: one execution of a kernel at a position in the launch grid. A kernel is written from the perspective of one instance operating on a block, or tile, of data. Many instances run the same compiled logic on different data. Triton manages the individual threads within the block; the corresponding CUDA term is block. section 1.3.3 section 2.1.1 section 2.2

Vector addition

For two vectors of eight elements with a block size of four, two instances do the work. The first adds elements 0–3; the second adds elements 4–7. Each performs a vectorized block-level operation, not necessarily a single hardware instruction. section 2.1.1

Figure 2.1 CUDA assigns one thread to each element, so two blocks of four threads (T0-T3) each compute four scalar additions individually. Triton assigns one program instance to each whole block, so program P0 computes z[0:4] = x[0:4] + y[0:4] and program P1 computes z[4:8] = x[4:8] + y[4:8] as single block-level operations.
CH02 F01 Fartale

Compare the two Triton instances with the individual CUDA threads assigned to the same eight additions. section 2.1.1

For a one-dimensional vector, the host can launch ceil(n_elements / BLOCK_SIZE) instances using triton.cdiv. Each instance gets its position with tl.program_id(0), computes block_start = pid * BLOCK_SIZE, and adds tl.arange(0, BLOCK_SIZE) to form element offsets. A mask checks offsets against n_elements so loads and stores skip positions beyond the end of the vector. section 2.2.3 section 2.2.8 section 2.2.9

A launch grid can have up to three dimensions; the axis passed to tl.program_id selects the coordinate to retrieve. The GPU schedules instances across streaming multiprocessors as resources become available. The compiler handles thread-level execution within each instance. section 2.2.3 section 2.4

The mapping from instances to data depends on the kernel. For example, a rotary-position-embedding kernel can assign one sequence row to each instance. If its grid has more entries than rows, it needs a row-bound guard before loading; a mask for feature positions alone does not protect an invalid row. For a reduction larger than one instance can cover, partial results can instead be reduced through successive launches. section 3.5.3 section 4.2.2

How GPU Programming with Triton covers it

How deep it goes
Works through grid sizing, program IDs, offsets, masking, and scheduling for a vector-addition kernel.
Examples
Uses vector addition and a rotary-position-embedding kernel to illustrate block and row assignments.
What you can do after
Apply it via Triton to assign vector slices to instances and mask out-of-range accesses.
How it connects
Connects program instances to launch grids, CUDA blocks, streaming multiprocessors, and reductions.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.