liveBook · Manning

Gradient checkpointing

As explained in Applied Reinforcement Learning

Gradient checkpointing reduces neural-network activation memory by recomputing activations during the backward pass instead of retaining them.

Gradient checkpointing lowers activation-memory use during training by recomputing activations during the backward pass instead of retaining them. For LoRA fine-tuning, gradients still flow through frozen layers to reach adapter parameters, so their activations matter too. The trade-off is more computation for less memory. section 11.3 section 11.5.3

Fitting training on one GPU

A 28-layer transformer processing 1,200 tokens can use over 10 GB for activations. Checkpointing recomputes them during backpropagation, adding roughly 30–40% compute time—a practical exchange when GPU memory is the bottleneck. section 11.5.3

How Applied Reinforcement Learning covers it

How deep it goes
Explains activation recomputation during backpropagation and its roughly 30–40% compute-time cost.
Examples
Quantifies activation use above 10 GB for a 28-layer transformer processing 1,200 tokens.
What it assumes
Assumes familiarity with LoRA adapters and backpropagation; does not teach either from first principles.
What you can do after
Operate/configure it with PyTorch’s gradient_checkpointing_enable interface to trade compute time for lower activation memory.
How it connects
Connects LoRA adapters, frozen transformer layers, backpropagation, and PyTorch.

Also explained in

Compare how 2 books explain gradient checkpointing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.