Memory hierarchy
A memory hierarchy is an arrangement of storage levels with differing capacities, access speeds, latencies, costs, or persistence.
GPU Programming with Triton
A memory hierarchy is an ordered arrangement of memory levels that trade capacity and access speed to keep frequently needed data near computation.
- How it is used
- Relates GPU memory organization to bandwidth use and latency hiding when reasoning about kernel performance.
- How deep it goes
- Describes five memory levels and their placement, and compares measured Hopper access latencies and H100 bandwidth characteristics.
- Examples
- Uses Hopper latency measurements and H100 memory organization, plus exercises ranking memory levels and inspecting a GPU.
- What you can do after
- Evaluate GPU memory characteristics and their implications for kernel performance.
Read how GPU Programming with Triton explains memory hierarchy →
Grokking GPU Programming
A memory hierarchy is an arrangement of storage levels with differing capacities, access latencies, and bandwidths.
- How it is used
- Organizes GPU storage choices around the tradeoff between capacity and access speed.
- How deep it goes
- Explains five levels, approximate latencies and capacities, L2 access, and shared-memory tradeoffs.
- Examples
- Uses a school analogy and a mountain diagram to illustrate storage levels and their tradeoffs.
- What you can do after
- Evaluate whether repeated data access justifies shared memory, and apply cooperative loading and block communication.
Read how Grokking GPU Programming explains memory hierarchy →
CUDA for LLMs
A memory hierarchy is an arrangement of storage levels with different capacities, access latencies, and sharing scopes.
- How it is used
- Guides CUDA performance optimization by prioritizing data residency in fast on-device tiers.
- How deep it goes
- Compares GPU tiers by capacity, latency, bandwidth, and sharing scope, with representative hardware figures.
- Examples
- Illustrates tiers with GPU latency and bandwidth estimates and identifies registers, shared memory, L1, L2, and global memory.
- What you can do after
- Optimize CUDA kernels by keeping data in the fastest feasible memory tier for as long as possible.
Fast Python: High performance techniques for large datasets
- How it is used
- Uses hardware hierarchy to explain performance in data processing, compressed storage, and computation over large arrays.
- How deep it goes
- Connects cache-aware array processing to storage and network levels, while keeping the architecture treatment selective.
- Examples
- Compares shuffled and unprocessed array representations by storage size and elapsed time.
- What you can do after
- Apply Blosc’s shuffle option to trade smaller representations against slightly longer processing time.
Read how Fast Python: High performance techniques for large datasets explains memory hierarchy →
Designing AI Agents
An organized set of memory stores with different capacities, persistence, and access costs.
- How it is used
- Organizes agent memory by scope, function, lifetime, access cost, and authority.
- How deep it goes
- Distinguishes memory tiers and governance dimensions, then identifies promotion, eviction, invalidation, and conflict failures.
- Examples
- Uses active instructions, task receipts, project conventions, archival traces, MemGPT paging, and Mem0 update operations.
- What you can do after
- Evaluate agent memory placement and authority, and retrieve the governing source before high-risk decisions.
Also covered in
What to know first
- threadA thread is a distinct sequence of execution with its own control flow and state, managed or scheduled by a processor, runtime, or operating system.
- hardware registerA hardware register is a small, rapidly accessible storage location within a processor or processing device that holds data, addresses, or control information during execution.
- CUDACUDA is NVIDIA’s parallel computing platform and programming model for performing general-purpose computations on NVIDIA GPUs.
Related concepts
- hardware registerhas part · A hardware register is a small, rapidly accessible storage location within a processor or processing device that holds data, addresses, or control information during execution.
- GPU memoryhas part · GPU memory is memory available to a graphics processing unit for storing data and state used in graphics or computation.
- shared memoryhas part · Shared memory is a memory region accessible to multiple processes, threads, or other concurrent execution contexts for sharing data.
- L1 cachehas part · An L1 cache is the smallest, lowest-latency level of a processor’s cache hierarchy, located closest to a processing core.
- L2 cachehas part · An L2 cache is a second-level hardware cache that stores copies of data to reduce memory-access costs and is typically larger and slower than L1 cache.
- CPU cachehas part · A CPU cache is small, fast memory that stores copies of data or instructions for quicker access by a processor.
Questions about memory hierarchy
What is memory hierarchy?
A memory hierarchy is an arrangement of storage levels with differing capacities, access speeds, latencies, costs, or persistence.
Which Manning books explain memory hierarchy?
GPU Programming with Triton, Grokking GPU Programming, CUDA for LLMs, Fast Python: High performance techniques for large datasets and Designing AI Agents.
What should I know before learning memory hierarchy?
Manning books that teach memory hierarchy usually introduce thread, hardware register, CUDA and disk first.
What are the parts of memory hierarchy?
Memory hierarchy includes hardware register, GPU memory, shared memory, L1 cache, L2 cache and CPU cache.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




