The distance problem is the delay caused by moving data from storage to a GPU’s arithmetic units. Global-memory access can take hundreds of clock cycles, leaving computation idle while data arrives. GPUs compensate by running many threads: while some wait, others compute. section 1.4
Figure 1.8 The memory hierarchy organized by speed and size. Registers top the hierarchy as the fastest and smallest storage; global memory sits at the bottom as the largest and slowest. Shared memory and the L1 cache appear on adjacent rows to reflect their distinct programming roles; on NVIDIA GPUs from the Volta architecture onwards, they share the same physical on-chip SRAM on each SM and have the same access latency, so the figure separates them by function, not by speed.
Compare the memory hierarchy’s storage levels by speed and size to see how data’s distance from the arithmetic units affects waiting. section 1.4
