Latency hiding keeps GPU hardware busy by scheduling other threads while some wait for memory data. Unlike approaches that try to make each access faster, GPUs use many concurrent threads to cover the wait. The technique depends on understanding the memory hierarchy and using its bandwidth efficiently. section 1.4
Figure 1.8 The memory hierarchy organized by speed and size. Registers top the hierarchy as the fastest and smallest storage; global memory sits at the bottom as the largest and slowest. Shared memory and the L1 cache appear on adjacent rows to reflect their distinct programming roles; on NVIDIA GPUs from the Volta architecture onwards, they share the same physical on-chip SRAM on each SM and have the same access latency, so the figure separates them by function, not by speed.
The hierarchy diagram shows the different storage levels that shape memory access and latency. section 1.4
