liveBook · Manning

Distance problem

As explained in GPU Programming with Triton

The performance problem caused by the latency of moving data from storage to computational units.

The distance problem is the delay caused by moving data from storage to a GPU’s arithmetic units. Global-memory access can take hundreds of clock cycles, leaving computation idle while data arrives. GPUs compensate by running many threads: while some wait, others compute. section 1.4

Figure 1.8 The memory hierarchy organized by speed and size. Registers top the hierarchy as the fastest and smallest storage; global memory sits at the bottom as the largest and slowest. Shared memory and the L1 cache appear on adjacent rows to reflect their distinct programming roles; on NVIDIA GPUs from the Volta architecture onwards, they share the same physical on-chip SRAM on each SM and have the same access latency, so the figure separates them by function, not by speed.
CH01 F07 Fartale

Compare the memory hierarchy’s storage levels by speed and size to see how data’s distance from the arithmetic units affects waiting. section 1.4

How GPU Programming with Triton covers it

How deep it goes
Defines memory latency and explains how concurrent threads help cover waiting, without detailing implementation.
Examples
Uses NVIDIA H100 global-memory latency and GPU clock cycles to illustrate the delay.
What it assumes
Defines clock cycles, while assuming familiarity with GPU memory and arithmetic units.
What you can do after
Understand why GPUs run many threads to keep computation active during memory delays.
How it connects
Connects memory latency, the GPU memory hierarchy, and concurrent threads.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.