liveBook · Manning

Memory hierarchy

A memory hierarchy is an arrangement of storage levels with differing capacities, access speeds, latencies, costs, or persistence.

How 5 Manning books explain it

GPU Programming with Triton

A memory hierarchy is an ordered arrangement of memory levels that trade capacity and access speed to keep frequently needed data near computation.

How it is used
Relates GPU memory organization to bandwidth use and latency hiding when reasoning about kernel performance.
How deep it goes
Describes five memory levels and their placement, and compares measured Hopper access latencies and H100 bandwidth characteristics.
Examples
Uses Hopper latency measurements and H100 memory organization, plus exercises ranking memory levels and inspecting a GPU.
What you can do after
Evaluate GPU memory characteristics and their implications for kernel performance.

Read how GPU Programming with Triton explains memory hierarchy →

Grokking GPU Programming

A memory hierarchy is an arrangement of storage levels with differing capacities, access latencies, and bandwidths.

How it is used
Organizes GPU storage choices around the tradeoff between capacity and access speed.
How deep it goes
Explains five levels, approximate latencies and capacities, L2 access, and shared-memory tradeoffs.
Examples
Uses a school analogy and a mountain diagram to illustrate storage levels and their tradeoffs.
What you can do after
Evaluate whether repeated data access justifies shared memory, and apply cooperative loading and block communication.

Read how Grokking GPU Programming explains memory hierarchy →

CUDA for LLMs

A memory hierarchy is an arrangement of storage levels with different capacities, access latencies, and sharing scopes.

How it is used
Guides CUDA performance optimization by prioritizing data residency in fast on-device tiers.
How deep it goes
Compares GPU tiers by capacity, latency, bandwidth, and sharing scope, with representative hardware figures.
Examples
Illustrates tiers with GPU latency and bandwidth estimates and identifies registers, shared memory, L1, L2, and global memory.
What you can do after
Optimize CUDA kernels by keeping data in the fastest feasible memory tier for as long as possible.

Read how CUDA for LLMs explains memory hierarchy →

Fast Python: High performance techniques for large datasets

How it is used
Uses hardware hierarchy to explain performance in data processing, compressed storage, and computation over large arrays.
How deep it goes
Connects cache-aware array processing to storage and network levels, while keeping the architecture treatment selective.
Examples
Compares shuffled and unprocessed array representations by storage size and elapsed time.
What you can do after
Apply Blosc’s shuffle option to trade smaller representations against slightly longer processing time.

Read how Fast Python: High performance techniques for large datasets explains memory hierarchy →

Designing AI Agents

An organized set of memory stores with different capacities, persistence, and access costs.

How it is used
Organizes agent memory by scope, function, lifetime, access cost, and authority.
How deep it goes
Distinguishes memory tiers and governance dimensions, then identifies promotion, eviction, invalidation, and conflict failures.
Examples
Uses active instructions, task receipts, project conventions, archival traces, MemGPT paging, and Mem0 update operations.
What you can do after
Evaluate agent memory placement and authority, and retrieve the governing source before high-risk decisions.

Read how Designing AI Agents explains memory hierarchy →

Also covered in

What to know first

Related concepts

Questions about memory hierarchy

What is memory hierarchy?

A memory hierarchy is an arrangement of storage levels with differing capacities, access speeds, latencies, costs, or persistence.

Which Manning books explain memory hierarchy?

GPU Programming with Triton, Grokking GPU Programming, CUDA for LLMs, Fast Python: High performance techniques for large datasets and Designing AI Agents.

What should I know before learning memory hierarchy?

Manning books that teach memory hierarchy usually introduce thread, hardware register, CUDA and disk first.

What are the parts of memory hierarchy?

Memory hierarchy includes hardware register, GPU memory, shared memory, L1 cache, L2 cache and CPU cache.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.