liveBook · Manning

Latency hiding

As explained in GPU Programming with Triton

A parallel execution technique that keeps processing units busy by running other threads while some threads wait for memory operations.

Latency hiding keeps GPU hardware busy by scheduling other threads while some wait for memory data. Unlike approaches that try to make each access faster, GPUs use many concurrent threads to cover the wait. The technique depends on understanding the memory hierarchy and using its bandwidth efficiently. section 1.4

Figure 1.8 The memory hierarchy organized by speed and size. Registers top the hierarchy as the fastest and smallest storage; global memory sits at the bottom as the largest and slowest. Shared memory and the L1 cache appear on adjacent rows to reflect their distinct programming roles; on NVIDIA GPUs from the Volta architecture onwards, they share the same physical on-chip SRAM on each SM and have the same access latency, so the figure separates them by function, not by speed.
CH01 F07 Fartale

The hierarchy diagram shows the different storage levels that shape memory access and latency. section 1.4

How GPU Programming with Triton covers it

How deep it goes
Defines the mechanism and rationale, then points to memory hierarchy without detailing implementation steps.
What you can do after
Understand why GPUs run many threads and how concurrent work can cover memory waits.
How it connects
Links GPU threads and memory hierarchy to concurrent execution, contrasting this approach with CPU memory-access strategies.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.