A hardware register is a small, fast storage location for a GPU thread’s active values. Each thread has a private set: arithmetic takes operands from registers and puts results back there. Loop counters, intermediate results, and frequently used values can all live in registers; other threads cannot see them. section 1.4 section 3.2.1
Registers sit at the top of the GPU memory hierarchy. Access takes roughly one clock cycle, while a trip to global memory can take hundreds. In a Triton kernel, tl.load brings values from global memory into on-chip registers for arithmetic, and tl.store writes results back. Masks protect out-of-range positions during loads and stores. section 1.4 section 2.2.7
Keep intermediates on chip
For z = x × sigmoid(x) × y, a fused kernel loads x and y once, computes the intermediate values in registers, and stores z once. With float32 values, this reduces global-memory traffic from 20 to 12 bytes per element compared with separate SiLU and multiplication kernels. The saving comes from avoiding the intermediate store and reload. section 3.3
Figure 3.4 A fused kernel loads both inputs once, performs the full SiLU and multiplication in registers, and writes the output once, eliminating the intermediate round-trip through global memory.
Follow the inputs and intermediate values through the fused kernel to see which global-memory transfers are eliminated. section 3.3
Registers are fast but limited. The register file on an NVIDIA H100 streaming multiprocessor (SM) contains 65,536 32-bit registers shared among its running threads. In a simplified allocation, 32 registers per thread permits 2,048 threads, assuming the SM supports that many; 64 per thread permits 1,024. Higher register use can therefore reduce occupancy and leave fewer threads available to hide memory latency. section 3.2.1 section 3.7
If a kernel needs more registers than are physically available, the compiler can spill excess values to local memory and reload them later. Local memory is backed by global DRAM and cached in L1 and L2; even a cache hit can be much slower than register access. section 3.2.1








