CPU cache

Q: How deeply does Fast Python: High performance techniques for large datasets cover CPU cache?

Works through cache hits, memory transfers, and contiguous matrix access, without detailing cache implementation.

As explained in Fast Python: High performance techniques for large datasets

A CPU cache is small, fast memory that stores copies of data or instructions for quicker access by a processor.

CPU caches sit between processor registers and main memory. A cache hit lets the processor use data without waiting for slower RAM; a miss can leave it idle while data is fetched. Cache capacity is limited, so the speed of a computation depends not just on its arithmetic but on which data it accesses and in what order. section 6.1.2

Table 6.1 Memory hierarchy with sizes and access times for a hypothetical, but realistic modern desktop (view table figure)

Type	Size	Access time
CPU
L1 cache	256 KB	2 ns
L2 cache	1 MB	5 ns
L3 cache	6 MB	30 ns
RAM
DIMM	8 GB	100 ns

Compare the illustrative capacities and access times of L1, L2, and L3 caches with RAM. The orders of magnitude matter more than the particular machine's numbers. section 6.1.2

Measure a NumPy access pattern

Create a square NumPy array, then time `2 * mat[:, 0]` and `2 * mat[0, :]` as its size grows. In one 10,000-by-10,000 test, doubling a column took 4.51 µs versus 74.9 µs for a row. The result is not a rule that columns are faster: the array's internal representation and the relationship between cache and main memory determine the difference. section 6.1.1 section 6.1.2

Accessing one matrix element can bring nearby elements into cache. Operations on contiguous elements may therefore reuse one memory transfer, while operations on separated elements may require a transfer for each access. Small arrays can conceal this cost; large arrays can make it dominant. Whether rows or columns are contiguous depends on the matrix representation. section 6.1.2

This also explains why extra computation is not always slower. A compressed block that fits in cache can sometimes be processed faster than raw data that repeatedly waits on RAM, even after accounting for decompression. Measure such gains on the intended machine: other processes can compete for cache, and NumExpr speedups observed on a server may shrink or disappear on a busy laptop. section 1.2.1 section 6.3.2

Table 6.6 The effect of hardware architecture on performance: Speedup as a function of hardware (view table figure)

Expression	Server speedup	Laptop speedup
a + a	3.8	0.7
f + f	3.8	0.8
a + f	3.7	1.3
a**5 + f	19.4	11.5
a**5 + f + sin(a) + cos(a)	25.1	6.7

The same NumExpr expressions show different speedups on a server and a laptop, illustrating why cache-sensitive optimizations need testing on the target hardware. section 6.3.2

How Fast Python: High performance techniques for large datasets covers it

Cover of Fast Python: High performance techniques for large datasets

liveBook manning.com

Read Fast Python: High performance techniques for large datasets on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

100 Go Mistakes and How to Avoid ThemA CPU cache is small, fast hardware memory that stores recently accessed data and instructions near a processor core.
liveBook manning.com

LatencyA CPU cache is a small, fast memory that stores copies of data likely to be accessed by a CPU.

Compare how 3 books explain CPU cache →

Related concepts

L1 cacheincludes · An L1 cache is a small, low-latency first-level cache located closest to a processor core or execution unit.
memory hierarchypart of · A memory hierarchy is an arrangement of storage levels with differing capacities, access speeds, latencies, costs, or persistence.
cache linehas part · A cache line is a fixed-size block of contiguous memory that a processor cache transfers and manages as a unit.
memorycontrasts with · Memory is a storage resource or mechanism that retains data or computational state for access during computation or later use.
data-oriented programmingused by · Data-oriented programming is an approach to structuring software around data representations, access patterns, and operations suited to the data's characteristics and intended use.

This explainer was generated by Manning's knowledge graph from the text of Fast Python: High performance techniques for large datasets. The section links point to the parts of the book it draws on.