liveBook · Manning

Hardware register

As explained in GPU Programming with Triton

A register is a small, private storage location associated with a GPU thread for operands, intermediate results, and other active values.

A hardware register is a small, fast storage location for a GPU thread’s active values. Each thread has a private set: arithmetic takes operands from registers and puts results back there. Loop counters, intermediate results, and frequently used values can all live in registers; other threads cannot see them. section 1.4 section 3.2.1

Registers sit at the top of the GPU memory hierarchy. Access takes roughly one clock cycle, while a trip to global memory can take hundreds. In a Triton kernel, tl.load brings values from global memory into on-chip registers for arithmetic, and tl.store writes results back. Masks protect out-of-range positions during loads and stores. section 1.4 section 2.2.7

Keep intermediates on chip

For z = x × sigmoid(x) × y, a fused kernel loads x and y once, computes the intermediate values in registers, and stores z once. With float32 values, this reduces global-memory traffic from 20 to 12 bytes per element compared with separate SiLU and multiplication kernels. The saving comes from avoiding the intermediate store and reload. section 3.3

Figure 3.4 A fused kernel loads both inputs once, performs the full SiLU and multiplication in registers, and writes the output once, eliminating the intermediate round-trip through global memory.
CH03 F04 Fartale

Follow the inputs and intermediate values through the fused kernel to see which global-memory transfers are eliminated. section 3.3

Registers are fast but limited. The register file on an NVIDIA H100 streaming multiprocessor (SM) contains 65,536 32-bit registers shared among its running threads. In a simplified allocation, 32 registers per thread permits 2,048 threads, assuming the SM supports that many; 64 per thread permits 1,024. Higher register use can therefore reduce occupancy and leave fewer threads available to hide memory latency. section 3.2.1 section 3.7

If a kernel needs more registers than are physically available, the compiler can spill excess values to local memory and reload them later. Local memory is backed by global DRAM and cached in L1 and L2; even a cache hit can be much slower than register access. section 3.2.1

How GPU Programming with Triton covers it

How deep it goes
Explains register allocation, occupancy trade-offs, and spilling to cached local memory.
Examples
Illustrates register use with vector addition, fused SiLU and multiplication, and fused softmax.
What you can do after
Evaluate how register usage and fusion affect memory traffic and the ability to hide latency.
How it connects
Connects registers to the GPU memory hierarchy, latency hiding, occupancy, and kernel fusion.

Also explained in

Compare how 9 books explain hardware register →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.