liveBook · Manning

Memory bound

As explained in Rearchitecting LLMs

Memory-bound describes a workload whose performance is limited primarily by memory transfers rather than arithmetic processing.

A workload is memory-bound when moving data, rather than performing arithmetic, limits performance. During LLM inference, GPUs can spend much of their time waiting for data from memory, leaving arithmetic capacity underused. section 4.1.2

Depth pruning targets this bottleneck by removing entire Transformer blocks. Each removed block eliminates a round of loading data from VRAM, processing it, and writing results back. It also reduces model memory and the KV-cache, which stores attention states for generated tokens. section 4.1.2 section 4.6

Removing layers reduces cache growth

Removing 25% of a model’s layers reduces KV-cache memory growth by 25% for each additional generated token. For Qwen3-06B, each Transformer block accounts for 2.1% of model size. section 4.1.2

How Rearchitecting LLMs covers it

How deep it goes
Explains memory transfers, GPU underuse, and KV-cache effects, without detailing hardware internals.
Examples
Uses Qwen3-06B block sizes and a 25% layer-removal example to illustrate memory reductions.
What you can do after
Apply depth pruning to reduce memory transfers and KV-cache growth during inference.
How it connects
Connects memory-bound inference to depth pruning, Transformer blocks, VRAM, and the KV-cache.

Also explained in

Compare how 18 books explain memory bound →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.