liveBook · Manning

Gradient checkpointing

Gradient checkpointing is a memory-saving training technique that stores selected intermediate activations and recomputes others during backpropagation.

How 2 Manning books explain it

LLM Customization and Fine-Tuning

A memory-saving training technique that discards intermediate activations and recomputes them as needed during backpropagation.

How it is used
Uses checkpointing to fit memory-constrained LoRA, full SFT, and DPO training runs.
How deep it goes
Explains forward-pass activation disposal and backward-pass recomputation, with estimated memory savings and compute costs.
Examples
Illustrates a full-parameter 4B DPO run peaking near 54 GB across three cards despite checkpointing.
What you can do after
Operate training configurations using checkpointing alongside small batches and gradient accumulation when activation memory is limiting.

Read how LLM Customization and Fine-Tuning explains gradient checkpointing →

Applied Reinforcement Learning

Gradient checkpointing reduces neural-network activation memory by recomputing activations during the backward pass instead of retaining them.

How it is used
Reduces activation-memory pressure during LoRA fine-tuning when GPU capacity is the bottleneck.
How deep it goes
Explains activation recomputation during backpropagation and its roughly 30–40% compute-time cost.
Examples
Quantifies activation use above 10 GB for a 28-layer transformer processing 1,200 tokens.
What you can do after
Operate/configure it with PyTorch’s gradient_checkpointing_enable interface to trade compute time for lower activation memory.

Read how Applied Reinforcement Learning explains gradient checkpointing →

Also covered in

What to know first

Related concepts

Questions about gradient checkpointing

What is gradient checkpointing?

Gradient checkpointing is a memory-saving training technique that stores selected intermediate activations and recomputes others during backpropagation.

Which Manning books explain gradient checkpointing?

LLM Customization and Fine-Tuning and Applied Reinforcement Learning.

What should I know before learning gradient checkpointing?

Manning books that teach gradient checkpointing usually introduce low-rank adaptation, adam, adapter and backpropagation first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.