A CPU cache hierarchy is a set of cache levels that trade capacity for access speed, keeping frequently needed code and data closer to the processor than DRAM. L1 is smallest and fastest; the last-level cache is larger and slower. When a working set misses in a cache, the processor fetches from a slower level or DRAM, increasing latency variance. section 2.3.2
Use the hierarchy as a practical low-latency tuning concern:
- Keep latency-sensitive working sets small enough for faster cache levels where feasible; the latest-generation L1 caches reach up to 200 KiB, while last-level caches reach tens of MiB. section 2.3.2
- Treat cache misses as a source of tail-latency variation, because software cannot directly control every hardware optimization affecting access time. section 2.3.2
- Account for multicore memory topology: on NUMA systems, colocate data with computation or partition application state across NUMA nodes. section 3.4
Latency-sensitive key–value service
A key–value store can partition its keyspace across NUMA nodes or CPU cores and forward each lookup to a nearby processor. This reduces remote-memory access while keeping requests closer to the data they use. section 3.4
Figure 3.4 UMA and NUMA architectures. The left side of the figure represents a UMA architecture where all CPU cores have the same access latency to DRAM. The right side represents a NUMA architecture with two nodes. CPUs that are in the same node can access NUMA node local memory faster than memory in a remote node.
Compare UMA, where cores see uniform DRAM access latency, with NUMA, where local memory is faster than remote memory. section 3.4
