liveBook · Manning

Memory usage

As explained in Rearchitecting LLMs

Memory consumption is the amount of computer memory required to store and execute a model or computation.

LLM inference uses GPU memory for model parameters and for the KV cache, which grows with context length. Memory-bound workloads can also spend time waiting for data to move between memory and the GPU. section 4.1.2 section 8.1

Pruning and quantization target different parts of the memory footprint:

  • Removing complete Transformer blocks reduces model size and eliminates those blocks’ contributions to KV-cache growth and memory traffic. section 4.1.2
  • Quantizing the KV cache reduces dynamic memory use, allowing larger prompts to be processed without running out of GPU memory. section 8.1
  • Attention pruning can reduce dynamic VRAM, but more pruning can sharply degrade benchmark scores and generation quality. section 8.4.2
Table 8.2 Dynamic VRAM delta across pruning configurations
Configuration
Dynamic VRAM
Vs Baseline
Baseline
109.57 MB
_
3 layers removed
96.06 MB
-12.3%
6 layers removed
78.72 MB
-28.2%
15 layers removed
58.63 MB
-46.5%

Compare dynamic VRAM against the baseline across three attention-pruning configurations. section 8.4.2

How Rearchitecting LLMs covers it

How deep it goes
Separates parameter and KV-cache savings, then reports pruning measurements and quality costs without giving a general memory-planning procedure.
Examples
Compares Qwen3 block removal and Llama attention-pruning configurations, including dynamic VRAM and benchmark changes.
What it assumes
Assumes prior exposure to weight quantization and model inference, while explaining why the KV cache grows with context.
What you can do after
Evaluate memory savings against benchmark and generation-quality losses, while treating shared-T4 throughput as directional.
How it connects
Links memory usage to depth pruning, attention pruning, KV-cache quantization, and inference speed.

Also explained in

Compare how 13 books explain memory usage →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.