Gradient checkpointing lowers activation-memory use during training by recomputing activations during the backward pass instead of retaining them. For LoRA fine-tuning, gradients still flow through frozen layers to reach adapter parameters, so their activations matter too. The trade-off is more computation for less memory. section 11.3 section 11.5.3
Fitting training on one GPU
A 28-layer transformer processing 1,200 tokens can use over 10 GB for activations. Checkpointing recomputes them during backpropagation, adding roughly 30–40% compute time—a practical exchange when GPU memory is the bottleneck. section 11.5.3

