Gradient checkpointing
Gradient checkpointing is a memory-saving training technique that stores selected intermediate activations and recomputes others during backpropagation.
LLM Customization and Fine-Tuning
A memory-saving training technique that discards intermediate activations and recomputes them as needed during backpropagation.
- How it is used
- Uses checkpointing to fit memory-constrained LoRA, full SFT, and DPO training runs.
- How deep it goes
- Explains forward-pass activation disposal and backward-pass recomputation, with estimated memory savings and compute costs.
- Examples
- Illustrates a full-parameter 4B DPO run peaking near 54 GB across three cards despite checkpointing.
- What you can do after
- Operate training configurations using checkpointing alongside small batches and gradient accumulation when activation memory is limiting.
Read how LLM Customization and Fine-Tuning explains gradient checkpointing →
Applied Reinforcement Learning
Gradient checkpointing reduces neural-network activation memory by recomputing activations during the backward pass instead of retaining them.
- How it is used
- Reduces activation-memory pressure during LoRA fine-tuning when GPU capacity is the bottleneck.
- How deep it goes
- Explains activation recomputation during backpropagation and its roughly 30–40% compute-time cost.
- Examples
- Quantifies activation use above 10 GB for a 28-layer transformer processing 1,200 tokens.
- What you can do after
- Operate/configure it with PyTorch’s gradient_checkpointing_enable interface to trade compute time for lower activation memory.
Read how Applied Reinforcement Learning explains gradient checkpointing →
Also covered in
- Crack Any Codebase with AIIn early access (MEAP)
- Sutskever's List
What to know first
- low-rank adaptationLow-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
- adamAdam is an adaptive stochastic-gradient optimization algorithm that updates parameters using exponential moving estimates of the first and second moments of gradients.
- adapterAn adapter is a small trainable module added to a pretrained model to modify its behavior while leaving most of the model’s original parameters unchanged.
- backpropagationBackpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
Related concepts
- GPipeused by · GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters.
- backpropagationuses · Backpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
- GPU memoryaddresses · GPU memory is memory available to a graphics processing unit for storing data and state used in graphics or computation.
- PyTorchuses · PyTorch is an open-source machine-learning framework that provides tensor computation, automatic differentiation, and tools for developing and training neural networks.
- activationaddresses · An activation is a numerical value or tensor produced by a neural-network unit, layer, or operation during a forward computation.
- GPUaddresses · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and other data-parallel workloads.
Questions about gradient checkpointing
What is gradient checkpointing?
Gradient checkpointing is a memory-saving training technique that stores selected intermediate activations and recomputes others during backpropagation.
Which Manning books explain gradient checkpointing?
LLM Customization and Fine-Tuning and Applied Reinforcement Learning.
What should I know before learning gradient checkpointing?
Manning books that teach gradient checkpointing usually introduce low-rank adaptation, adam, adapter and backpropagation first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

